[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:37 dist-test-slave-zw07 krb5kdc[4291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:37 dist-test-slave-zw07 krb5kdc[4291](info): set up 2 sockets
Sep 25 07:20:37 dist-test-slave-zw07 krb5kdc[4291](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 07:20:38.192224 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:38.313653 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:42561
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:38.393990 4304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:38.394244 4304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:38.394279 4304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:38.395408 4304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:38.395486 4304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:38.395510 4304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:38.395539 4304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:38.395560 4304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:38.396996 4304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:38.397274 4304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:38.397495 4304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:20:38.402736 4304 server_base.cc:1034] running on GCE node
W20260925 07:20:38.402681 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.402693 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.402684 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:38.403244 4304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:38.403504 4304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:38.404656 4304 hybrid_clock.cc:648] HybridClock initialized: now 1790320838404636 us; error 41 us; skew 500 ppm
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:38.415007 4304 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:38.415596 4304 webserver.cc:460] Webserver started at http://127.4.47.62:41287/ using document root <none> and password file <none>
I20260925 07:20:38.415833 4304 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:38.415879 4304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:38.416054 4304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:38.417176 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "c7f4177575344f5eade8a0f775fd256b"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "c30365e90c8f189ebde5f8f1872176f0"
server_key_iv: "90ed914634f961e75e6a640d46dc07f9"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.417591 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "c7f4177575344f5eade8a0f775fd256b"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "c30365e90c8f189ebde5f8f1872176f0"
server_key_iv: "90ed914634f961e75e6a640d46dc07f9"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.418937 4304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:38.419839 4319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.420064 4304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:20:38.420156 4304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c7f4177575344f5eade8a0f775fd256b"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "c30365e90c8f189ebde5f8f1872176f0"
server_key_iv: "90ed914634f961e75e6a640d46dc07f9"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.420220 4304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:38.429229 4304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:38.430984 4304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:38.431159 4304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:38.434788 4304 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:42561
I20260925 07:20:38.434837 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:42561 every 8 connection(s)
I20260925 07:20:38.435243 4304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:38.436120 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:38.438287 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b: Bootstrap starting.
I20260925 07:20:38.438659 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4304
I20260925 07:20:38.438743 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:38.438927 4372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:38.439273 4372 log.cc:824] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:38.439489 4284 external_mini_cluster.cc:1428] Setting key e9294fc326a532b497cfd2dbad0b5cda
I20260925 07:20:38.440062 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b: No bootstrap required, opened a new log
I20260925 07:20:38.441892 4372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } }
I20260925 07:20:38.442058 4372 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:38.442108 4372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f4177575344f5eade8a0f775fd256b, State: Initialized, Role: FOLLOWER
I20260925 07:20:38.442240 4372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } }
I20260925 07:20:38.442337 4372 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:38.442396 4372 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:38.442459 4372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:38.443010 4372 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } }
I20260925 07:20:38.443150 4372 leader_election.cc:302] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f4177575344f5eade8a0f775fd256b; no voters:
I20260925 07:20:38.443305 4372 leader_election.cc:288] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:38.443333 4377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:38.443446 4377 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [term 1 LEADER]: Becoming Leader. State: Replica: c7f4177575344f5eade8a0f775fd256b, State: Running, Role: LEADER
I20260925 07:20:38.443554 4372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:38.443576 4377 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } }
I20260925 07:20:38.443967 4377 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f4177575344f5eade8a0f775fd256b. Latest consensus state: current_term: 1 leader_uuid: "c7f4177575344f5eade8a0f775fd256b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } } }
I20260925 07:20:38.444232 4377 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:38.444152 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f4177575344f5eade8a0f775fd256b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f4177575344f5eade8a0f775fd256b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42561 } } }
I20260925 07:20:38.444406 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:38.444794 4386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:38.445353 4386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:38.448457 4386 catalog_manager.cc:1347] Generated new cluster ID: c9396af040d64171b69dd0ee83e0c4f5
I20260925 07:20:38.448509 4386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:38.455699 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.440430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.4.47.62:42561)
0925 07:20:38.440790 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:38.440798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:38.440840 (+ 42us) server_negotiation.cc:407] Connection header received
0925 07:20:38.442497 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:38.442503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:38.442555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:38.442711 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:38.445032 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.445989 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.447321 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.447915 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.448033 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.451571 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:38.451576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:38.451576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:38.451588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:38.453780 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.454226 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.454229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.454229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.454335 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.454613 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.454613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.454614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.454849 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:38.455015 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:38.455333 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 07:20:38.455467 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":66,"threads_started":1}
I20260925 07:20:38.468715 4386 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:38.469293 4386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:38.476073 4386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f4177575344f5eade8a0f775fd256b: Generated new TSK 0
I20260925 07:20:38.476280 4386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:38.511422 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:38.592705 4398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:38.592947 4398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:38.592986 4398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:38.594165 4398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:38.594240 4398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:38.594288 4398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:38.595795 4398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:38.596058 4398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:38.596287 4398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:20:38.601323 4398 server_base.cc:1034] running on GCE node
W20260925 07:20:38.601301 4403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.601354 4404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.601301 4406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:38.601811 4398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:38.602066 4398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:38.603235 4398 hybrid_clock.cc:648] HybridClock initialized: now 1790320838603197 us; error 48 us; skew 500 ppm
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:38.605751 4398 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:38.606240 4398 webserver.cc:460] Webserver started at http://127.4.47.1:44971/ using document root <none> and password file <none>
I20260925 07:20:38.606463 4398 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:38.606530 4398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:38.606684 4398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:38.607801 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "e6f63f8884254b8297d72b5718f73087"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "6af51493c48b4d6cd2dba7d2bb11688a"
server_key_iv: "f5731bcd09e146c8f52064a8070be25a"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.608167 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "e6f63f8884254b8297d72b5718f73087"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "6af51493c48b4d6cd2dba7d2bb11688a"
server_key_iv: "f5731bcd09e146c8f52064a8070be25a"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.609486 4398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:38.610217 4413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.610380 4398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:38.610481 4398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "e6f63f8884254b8297d72b5718f73087"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "6af51493c48b4d6cd2dba7d2bb11688a"
server_key_iv: "f5731bcd09e146c8f52064a8070be25a"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.610556 4398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:38.621948 4398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:38.624106 4398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:38.624271 4398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:38.624505 4398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:38.624833 4398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:38.624887 4398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.624926 4398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:38.624950 4398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.630885 4398 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:36441
I20260925 07:20:38.630934 4526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:36441 every 8 connection(s)
I20260925 07:20:38.631309 4398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:38.636487 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4398
I20260925 07:20:38.636605 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:38.636801 4284 external_mini_cluster.cc:1428] Setting key 40df3eb9eea16746f8f18df8913b42a0
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:38.641327 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.632193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:38519 (local address 127.4.47.62:42561)
0925 07:20:38.632256 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:38.632259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:38.632488 (+ 229us) server_negotiation.cc:407] Connection header received
0925 07:20:38.632989 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:38.632990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:38.633003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:38.633109 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:38.634086 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.634590 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.635591 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.635800 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.635862 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.638323 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:38.638324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:38.638325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:38.638332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:38.639870 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.640317 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.640318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.640319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.640378 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.640645 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.640646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.640646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.640760 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:38.640831 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:38.641138 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 07:20:38.641228 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:20:38.641525 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.632182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42561 (local address 127.4.47.1:38519)
0925 07:20:38.632355 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:38.632391 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:38.632868 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:38.633170 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:38.633173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:38.633750 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:38.633947 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.633951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.634679 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.634680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.635508 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.635510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.635879 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.635880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.635984 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.636536 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:38.636547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:38.638212 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:38.639973 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.639980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.639983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.640225 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.640443 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.640444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.640445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.640569 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.640836 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:38.640842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:38.640988 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 07:20:38.641118 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 07:20:38.641283 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260925 07:20:38.642014 4527 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42561
I20260925 07:20:38.642114 4527 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:38.642297 4527 heartbeater.cc:511] Master 127.4.47.62:42561 requested a full tablet report, sending...
I20260925 07:20:38.642908 4336 ts_manager.cc:194] Registered new tserver with Master: e6f63f8884254b8297d72b5718f73087 (127.4.47.1:36441)
I20260925 07:20:38.643893 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:38519
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:38.663048 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:38.746888 4532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:38.747146 4532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:38.747195 4532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:38.748346 4532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:38.748387 4532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:38.748442 4532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:38.750440 4532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:38.750702 4532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:38.750955 4532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:38.755929 4537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.756012 4538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.755929 4540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:38.756034 4532 server_base.cc:1034] running on GCE node
I20260925 07:20:38.756319 4532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:38.756569 4532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:38.757707 4532 hybrid_clock.cc:648] HybridClock initialized: now 1790320838757697 us; error 32 us; skew 500 ppm
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:38.760334 4532 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:38.760802 4532 webserver.cc:460] Webserver started at http://127.4.47.2:42045/ using document root <none> and password file <none>
I20260925 07:20:38.761049 4532 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:38.761096 4532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:38.761226 4532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:38.762262 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "d0b3d7d8216f42c885d2db78f59a81e0"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "4659aa0cd205f0a1b1002977767fd849"
server_key_iv: "5465593f1df0a8621c7d37dbad08c2b3"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.762617 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "d0b3d7d8216f42c885d2db78f59a81e0"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "4659aa0cd205f0a1b1002977767fd849"
server_key_iv: "5465593f1df0a8621c7d37dbad08c2b3"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.764005 4532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:38.764747 4547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.764920 4532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:38.765017 4532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "d0b3d7d8216f42c885d2db78f59a81e0"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "4659aa0cd205f0a1b1002977767fd849"
server_key_iv: "5465593f1df0a8621c7d37dbad08c2b3"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.765096 4532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:38.785760 4532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:38.787885 4532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:38.788056 4532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:38.788306 4532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:38.788619 4532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:38.788674 4532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.788707 4532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:38.788735 4532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.794989 4532 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:40281
I20260925 07:20:38.795037 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:40281 every 8 connection(s)
I20260925 07:20:38.795435 4532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:38.797927 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4532
I20260925 07:20:38.798039 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:38.798213 4284 external_mini_cluster.cc:1428] Setting key 6c738026f82fda8b9b2a035d5c55f263
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:38.806053 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.796185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:50927 (local address 127.4.47.62:42561)
0925 07:20:38.796252 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:38.796254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:38.796431 (+ 177us) server_negotiation.cc:407] Connection header received
0925 07:20:38.796904 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:38.796904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:38.796914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:38.796981 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:38.798021 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.798717 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.800394 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.800734 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.800816 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.802976 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:38.802978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:38.802978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:38.802987 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:38.804688 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.805066 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.805067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.805067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.805134 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.805374 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.805375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.805375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.805518 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:38.805594 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:38.805849 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 07:20:38.805940 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:20:38.806080 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.796155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42561 (local address 127.4.47.2:50927)
0925 07:20:38.796303 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:38.796345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:38.796822 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:38.797039 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:38.797043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:38.797678 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:38.797913 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.797918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.798911 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.798912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.800289 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.800293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.800822 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.800823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.800927 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.801531 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:38.801539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:38.802879 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:38.804773 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.804778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.804781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.804988 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.805209 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.805210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.805210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.805306 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.805588 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:38.805591 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:38.805718 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 07:20:38.805800 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:20:38.805919 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 07:20:38.806690 4661 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42561
I20260925 07:20:38.806782 4661 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
I20260925 07:20:38.806995 4661 heartbeater.cc:511] Master 127.4.47.62:42561 requested a full tablet report, sending...
I20260925 07:20:38.807471 4336 ts_manager.cc:194] Registered new tserver with Master: d0b3d7d8216f42c885d2db78f59a81e0 (127.4.47.2:40281)
I20260925 07:20:38.807813 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:50927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:38.826915 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:38.908871 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:38.909088 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:38.909112 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:38.910180 4666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:38.910215 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:38.910241 4666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:38.911756 4666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:38.911939 4666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:38.912119 4666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:38.917053 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:38.917076 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:38.917301 4666 server_base.cc:1034] running on GCE node
W20260925 07:20:38.917060 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:38.917542 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:38.917805 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:38.918987 4666 hybrid_clock.cc:648] HybridClock initialized: now 1790320838918971 us; error 45 us; skew 500 ppm
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:38.921582 4666 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:38.922083 4666 webserver.cc:460] Webserver started at http://127.4.47.3:34083/ using document root <none> and password file <none>
I20260925 07:20:38.922331 4666 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:38.922400 4666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:38.922566 4666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:38.923750 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "891d652a79e046f8a5c4e786e06af1f6"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "bcf6b3266536acde401ae0d4b11a5e5c"
server_key_iv: "637a6cf872b64f0dbd810615c877b3be"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.924185 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "891d652a79e046f8a5c4e786e06af1f6"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "bcf6b3266536acde401ae0d4b11a5e5c"
server_key_iv: "637a6cf872b64f0dbd810615c877b3be"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.925526 4666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:20:38.926414 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.926667 4666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:38.926786 4666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "891d652a79e046f8a5c4e786e06af1f6"
format_stamp: "Formatted at 2026-09-25 07:20:38 on dist-test-slave-zw07"
server_key: "bcf6b3266536acde401ae0d4b11a5e5c"
server_key_iv: "637a6cf872b64f0dbd810615c877b3be"
server_key_version: "encryptionkey@0"
I20260925 07:20:38.926863 4666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:38.934659 4666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:38.936717 4666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:38.936880 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:38.937117 4666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:38.937429 4666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:38.937485 4666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.937528 4666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:38.937557 4666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:38.943800 4666 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:41495
I20260925 07:20:38.943863 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:41495 every 8 connection(s)
I20260925 07:20:38.944211 4666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:38.952113 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4666
I20260925 07:20:38.952242 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:38.952473 4284 external_mini_cluster.cc:1428] Setting key 96dc990c4f1c86f46a30cafe9b307476
I20260925 07:20:38.954401 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.945022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:49637 (local address 127.4.47.62:42561)
0925 07:20:38.945067 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:38.945069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:38.945323 (+ 254us) server_negotiation.cc:407] Connection header received
0925 07:20:38.945863 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:38.945864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:38.945876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:38.945926 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:38.946880 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.947487 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.948499 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.948713 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.948791 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.950951 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:38.950953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:38.950953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:38.950960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:38.952986 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.953458 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.953459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.953460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.953526 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.953802 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.953803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.953803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.953935 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:38.953989 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:38.954275 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 07:20:38.954338 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:20:38.954603 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.945019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42561 (local address 127.4.47.3:49637)
0925 07:20:38.945183 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:38.945235 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:38.945748 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:38.945996 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:38.946000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:38.946569 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:38.946767 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.946771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.947582 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.947583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.948389 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:38.948391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.948799 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.948800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:38.948925 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.949413 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:38.949420 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:38.950850 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:38.953075 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.953082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.953085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.953362 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.953577 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:38.953578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:38.953578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:38.953736 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:38.953999 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:38.954004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:38.954123 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 07:20:38.954233 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 07:20:38.954359 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260925 07:20:38.955140 4795 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42561
I20260925 07:20:38.955235 4795 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:38.955377 4795 heartbeater.cc:511] Master 127.4.47.62:42561 requested a full tablet report, sending...
I20260925 07:20:38.955698 4336 ts_manager.cc:194] Registered new tserver with Master: 891d652a79e046f8a5c4e786e06af1f6 (127.4.47.3:41495)
I20260925 07:20:38.956128 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:49637
I20260925 07:20:38.964756 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:38.988390 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:38.980362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.4.47.1:36441)
0925 07:20:38.980535 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:38.980543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:38.980561 (+ 18us) server_negotiation.cc:407] Connection header received
0925 07:20:38.980592 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:38.980594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:38.980611 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:38.980680 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:38.981254 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.982019 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.982771 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:38.983219 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:38.983278 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:38.985580 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:38.985588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:38.985589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:38.985595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:38.987316 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.987669 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.987672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.987672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.987724 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:38.987946 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:38.987946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:38.987946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:38.988072 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:38.988176 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:38.988251 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:20:38.988311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
W20260925 07:20:38.988782 4441 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37202
Sep 25 07:20:38 dist-test-slave-zw07 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:20:39 dist-test-slave-zw07 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:39.010983 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:39.003845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.4.47.1:36441)
0925 07:20:39.003939 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:39.003940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:39.004006 (+ 66us) server_negotiation.cc:407] Connection header received
0925 07:20:39.004074 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:39.004075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:39.004089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:39.004137 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:39.004742 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:39.005231 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:39.005923 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:39.006114 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:39.006166 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:39.008475 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:39.008477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:39.008477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:39.008484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:39.009977 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:39.010307 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:39.010309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:39.010309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:39.010361 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:39.010526 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:39.010527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:39.010527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:39.010620 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:39.010721 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:39.010795 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 07:20:39.010867 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 07:20:39.011582 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4398
I20260925 07:20:39.018095 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4532
I20260925 07:20:39.022917 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4666
I20260925 07:20:39.027473 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4304
2026-09-25T07:20:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:39 dist-test-slave-zw07 krb5kdc[4809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:39 dist-test-slave-zw07 krb5kdc[4809](info): set up 2 sockets
Sep 25 07:20:39 dist-test-slave-zw07 krb5kdc[4809](info): commencing operation
krb5kdc: starting...
W20260925 07:20:40.074766 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:40.195020 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36551
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:40.276661 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:40.276885 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:40.276921 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:40.278115 4822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:40.278191 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:40.278216 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:40.278249 4822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:40.278275 4822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:40.279826 4822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:40.280100 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:40.280329 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:40.285284 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.285382 4822 server_base.cc:1034] running on GCE node
W20260925 07:20:40.285274 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:40.285274 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.285820 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:40.286082 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:40.287248 4822 hybrid_clock.cc:648] HybridClock initialized: now 1790320840287229 us; error 35 us; skew 500 ppm
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:40.289776 4822 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:40.290285 4822 webserver.cc:460] Webserver started at http://127.4.47.62:35625/ using document root <none> and password file <none>
I20260925 07:20:40.290537 4822 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:40.290606 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:40.290758 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:40.291901 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "8eab9603de144cfcb0b3b7831b8f7afd"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "05ae0e2105ae5b8565a7e38308f34a19"
server_key_iv: "75513a8a4e5aacb701e68fa240e5c315"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.292275 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "8eab9603de144cfcb0b3b7831b8f7afd"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "05ae0e2105ae5b8565a7e38308f34a19"
server_key_iv: "75513a8a4e5aacb701e68fa240e5c315"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.293610 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:40.294354 4837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.294631 4822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:40.294737 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "8eab9603de144cfcb0b3b7831b8f7afd"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "05ae0e2105ae5b8565a7e38308f34a19"
server_key_iv: "75513a8a4e5aacb701e68fa240e5c315"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.294811 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:40.305503 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:40.307689 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:40.307853 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:40.312093 4822 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:36551
I20260925 07:20:40.312152 4889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:36551 every 8 connection(s)
I20260925 07:20:40.312507 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:40.313513 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:40.315886 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd: Bootstrap starting.
I20260925 07:20:40.316474 4890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:40.316768 4890 log.cc:824] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:40.317472 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd: No bootstrap required, opened a new log
I20260925 07:20:40.319043 4890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } }
I20260925 07:20:40.319252 4890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:40.319276 4890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eab9603de144cfcb0b3b7831b8f7afd, State: Initialized, Role: FOLLOWER
I20260925 07:20:40.319434 4890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } }
I20260925 07:20:40.319502 4890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:40.319561 4890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:40.319608 4890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:40.319851 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4822
I20260925 07:20:40.319942 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:40.320101 4284 external_mini_cluster.cc:1428] Setting key 2f84240b2f8471af4f8dc9a922d96033
I20260925 07:20:40.320395 4890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } }
I20260925 07:20:40.320514 4890 leader_election.cc:302] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eab9603de144cfcb0b3b7831b8f7afd; no voters:
I20260925 07:20:40.320695 4890 leader_election.cc:288] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:40.320945 4890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:40.320922 4894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:40.321110 4894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [term 1 LEADER]: Becoming Leader. State: Replica: 8eab9603de144cfcb0b3b7831b8f7afd, State: Running, Role: LEADER
I20260925 07:20:40.321224 4894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } }
I20260925 07:20:40.321571 4894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eab9603de144cfcb0b3b7831b8f7afd. Latest consensus state: current_term: 1 leader_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } } }
I20260925 07:20:40.321583 4896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eab9603de144cfcb0b3b7831b8f7afd" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36551 } } }
I20260925 07:20:40.321660 4894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:40.321712 4896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:40.322494 4904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:40.323027 4904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:40.324571 4904 catalog_manager.cc:1347] Generated new cluster ID: 831a9987285f4355b333d3ba53bc6a86
I20260925 07:20:40.324626 4904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:40.331826 4904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:40.332583 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.320589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.4.47.62:36551)
0925 07:20:40.320819 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.320829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.320865 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:20:40.321084 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.321092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.321155 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.321276 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:40.322472 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.323841 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.325090 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.325735 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.325865 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.328649 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:40.328654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:40.328654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:40.328666 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:40.330706 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.331130 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.331133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.331133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.331243 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.331479 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.331480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.331481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.331782 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:40.331944 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.332206 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.332349 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260925 07:20:40.332620 4904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:40.336871 4904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eab9603de144cfcb0b3b7831b8f7afd: Generated new TSK 0
I20260925 07:20:40.337061 4904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:40.362320 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:40.444133 4916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:40.444396 4916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:40.444434 4916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:40.445639 4916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:40.445716 4916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:40.445765 4916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:40.447330 4916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:40.447599 4916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:40.447846 4916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:40.453724 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:40.453737 4922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.453873 4916 server_base.cc:1034] running on GCE node
W20260925 07:20:40.454008 4924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.454231 4916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:40.454507 4916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:40.455679 4916 hybrid_clock.cc:648] HybridClock initialized: now 1790320840455655 us; error 40 us; skew 500 ppm
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:40.458410 4916 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:40.458897 4916 webserver.cc:460] Webserver started at http://127.4.47.1:46533/ using document root <none> and password file <none>
I20260925 07:20:40.459172 4916 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:40.459249 4916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:40.459401 4916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:40.460454 4916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "54fcb56a354b49b7b9371b548cd499a5"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "d4cf16bb4afd38c9978e4bce972b4376"
server_key_iv: "2a3b8f7b3b20d233fa55dcfdcfbf49b1"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.460820 4916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "54fcb56a354b49b7b9371b548cd499a5"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "d4cf16bb4afd38c9978e4bce972b4376"
server_key_iv: "2a3b8f7b3b20d233fa55dcfdcfbf49b1"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.462136 4916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:40.462836 4931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.463004 4916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:40.463137 4916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "54fcb56a354b49b7b9371b548cd499a5"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "d4cf16bb4afd38c9978e4bce972b4376"
server_key_iv: "2a3b8f7b3b20d233fa55dcfdcfbf49b1"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.463227 4916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:40.475241 4916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:40.477139 4916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:40.477314 4916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:40.477576 4916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:40.477891 4916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:40.477944 4916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.477984 4916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:40.478014 4916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.483996 4916 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:42225
I20260925 07:20:40.484037 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:42225 every 8 connection(s)
I20260925 07:20:40.484356 4916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:40.487540 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 4916
I20260925 07:20:40.487619 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:40.487820 4284 external_mini_cluster.cc:1428] Setting key fee53c9160d712e3bda461e4bd01695c
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:40.494212 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.485071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:50917 (local address 127.4.47.62:36551)
0925 07:20:40.485114 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.485116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.485370 (+ 254us) server_negotiation.cc:407] Connection header received
0925 07:20:40.486371 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.486372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.486390 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.486447 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:40.487097 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.487669 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.488937 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.489137 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.489214 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.491375 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:40.491377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:40.491377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:40.491385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:40.492801 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.493250 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.493252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.493252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.493310 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.493561 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.493562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.493562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.493684 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:40.493763 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.494032 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.494114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:20:40.494422 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.485072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36551 (local address 127.4.47.1:50917)
0925 07:20:40.485253 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.485289 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.486247 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.486542 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.486545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.486797 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:40.486987 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.486991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.487757 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.487759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.488833 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.488835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.489220 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.489221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.489351 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.489880 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:40.489889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:40.491282 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:40.492902 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.492908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.492910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.493156 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.493376 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.493378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.493378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.493492 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.493752 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:40.493758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:40.493901 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.494012 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.494184 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260925 07:20:40.494907 5045 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36551
I20260925 07:20:40.495002 5045 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:40.495218 5045 heartbeater.cc:511] Master 127.4.47.62:36551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
I20260925 07:20:40.495724 4854 ts_manager.cc:194] Registered new tserver with Master: 54fcb56a354b49b7b9371b548cd499a5 (127.4.47.1:42225)
I20260925 07:20:40.496330 4854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:50917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:40.514516 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:40.597123 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:40.597334 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:40.597360 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:40.598644 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:40.598686 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:40.598713 5050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:40.600318 5050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:40.600586 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:40.600833 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:40.606124 5058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:40.606176 5056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.606345 5050 server_base.cc:1034] running on GCE node
W20260925 07:20:40.606437 5055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.606683 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:40.606945 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:40.608117 5050 hybrid_clock.cc:648] HybridClock initialized: now 1790320840608107 us; error 62 us; skew 500 ppm
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:40.610764 5050 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:40.611334 5050 webserver.cc:460] Webserver started at http://127.4.47.2:33323/ using document root <none> and password file <none>
I20260925 07:20:40.611562 5050 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:40.611622 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:40.611750 5050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:40.612792 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "83bee9435cf94a018f49e8b510bc9899"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "23af6215a7aa98109979209e325ae3b2"
server_key_iv: "987258ff8ffcf8dcb4430c5603f23f94"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.613157 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "83bee9435cf94a018f49e8b510bc9899"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "23af6215a7aa98109979209e325ae3b2"
server_key_iv: "987258ff8ffcf8dcb4430c5603f23f94"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.614488 5050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:40.615352 5065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.615532 5050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:40.615622 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "83bee9435cf94a018f49e8b510bc9899"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "23af6215a7aa98109979209e325ae3b2"
server_key_iv: "987258ff8ffcf8dcb4430c5603f23f94"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.615697 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:40.645097 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:40.647207 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:40.647358 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:40.647596 5050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:40.647918 5050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:40.647953 5050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.647974 5050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:40.648020 5050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.653775 5050 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:38201
I20260925 07:20:40.653843 5178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:38201 every 8 connection(s)
I20260925 07:20:40.654150 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:40.659837 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5050
I20260925 07:20:40.659919 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:40.660089 4284 external_mini_cluster.cc:1428] Setting key 0985483f8d80b23ab3530ab41870c998
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:40.664354 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.654961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:33825 (local address 127.4.47.62:36551)
0925 07:20:40.655044 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.655047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.655263 (+ 216us) server_negotiation.cc:407] Connection header received
0925 07:20:40.656280 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.656282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.656295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.656354 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:40.657002 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.657503 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.658526 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.658721 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.658797 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.661253 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:40.661254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:40.661254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:40.661260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:40.662855 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.663383 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.663383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.663384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.663435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.663725 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.663726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.663726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.663831 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:40.663880 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.664226 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.664291 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:20:40.664642 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.654903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36551 (local address 127.4.47.2:33825)
0925 07:20:40.655119 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.655166 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.656131 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.656428 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.656433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.656715 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:40.656902 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.656906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.657599 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.657600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.658423 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.658425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.658802 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.658804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.658935 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.659512 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:40.659520 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:40.661151 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:40.662935 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.662941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.662943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.663306 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.663502 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.663503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.663503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.663649 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.663893 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:40.663899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:40.664070 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.664187 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.664364 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260925 07:20:40.665159 5179 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36551
I20260925 07:20:40.665273 5179 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:40.665484 5179 heartbeater.cc:511] Master 127.4.47.62:36551 requested a full tablet report, sending...
I20260925 07:20:40.665885 4854 ts_manager.cc:194] Registered new tserver with Master: 83bee9435cf94a018f49e8b510bc9899 (127.4.47.2:38201)
I20260925 07:20:40.666278 4854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:33825
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:40.689949 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:40.776137 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:40.776348 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:40.776372 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:40.777546 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:40.777591 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:40.777618 5184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:40.779151 5184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:40.779392 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:40.779613 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:40.784726 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:40.784745 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:40.784747 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:40.784960 5184 server_base.cc:1034] running on GCE node
I20260925 07:20:40.785288 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:40.785526 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:40.786700 5184 hybrid_clock.cc:648] HybridClock initialized: now 1790320840786686 us; error 29 us; skew 500 ppm
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:40.789296 5184 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:40.789819 5184 webserver.cc:460] Webserver started at http://127.4.47.3:34177/ using document root <none> and password file <none>
I20260925 07:20:40.790004 5184 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:40.790045 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:40.790135 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:40.791205 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "858e41fbf6514476b5e42feda4802e38"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "217163bcbc73e9b24cde825cacfa8593"
server_key_iv: "e694e2d2348d630111a394f758caff25"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.791517 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "858e41fbf6514476b5e42feda4802e38"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "217163bcbc73e9b24cde825cacfa8593"
server_key_iv: "e694e2d2348d630111a394f758caff25"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.792855 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:40.793783 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.794147 5184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:40.794246 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "858e41fbf6514476b5e42feda4802e38"
format_stamp: "Formatted at 2026-09-25 07:20:40 on dist-test-slave-zw07"
server_key: "217163bcbc73e9b24cde825cacfa8593"
server_key_iv: "e694e2d2348d630111a394f758caff25"
server_key_version: "encryptionkey@0"
I20260925 07:20:40.794325 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:40.815840 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:40.818181 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:40.818305 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:40.818620 5184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:40.818946 5184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:40.819000 5184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.819046 5184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:40.819092 5184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:40.825479 5184 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:41107
I20260925 07:20:40.825523 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:41107 every 8 connection(s)
I20260925 07:20:40.825852 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:20:40 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:40.835166 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5184
I20260925 07:20:40.835265 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:40.835270 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.826589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:46513 (local address 127.4.47.62:36551)
0925 07:20:40.826651 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.826653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.826868 (+ 215us) server_negotiation.cc:407] Connection header received
0925 07:20:40.827867 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.827868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.827879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.827943 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:40.828622 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.829155 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.830127 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.830309 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.830365 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.832498 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:40.832501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:40.832501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:40.832512 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:40.833934 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.834313 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.834314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.834314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.834363 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.834621 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.834622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.834622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.834720 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:40.834770 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.835058 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.835159 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:20:40.835449 4284 external_mini_cluster.cc:1428] Setting key 0b5b49969659c39866f4a87686d0afb9
I20260925 07:20:40.835520 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.826591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36551 (local address 127.4.47.3:46513)
0925 07:20:40.826737 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.826779 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.827767 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.828016 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.828020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.828344 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:40.828533 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.828538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.829237 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.829237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.830036 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.830038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.830383 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.830383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.830479 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.830981 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:40.830990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:40.832400 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:40.834018 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.834023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.834025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.834248 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.834440 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:40.834441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:40.834442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:40.834552 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:40.834781 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:40.834785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:40.834901 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.834990 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.835213 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 07:20:40.836108 5313 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36551
I20260925 07:20:40.836211 5313 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:40.836403 5313 heartbeater.cc:511] Master 127.4.47.62:36551 requested a full tablet report, sending...
I20260925 07:20:40.836766 4854 ts_manager.cc:194] Registered new tserver with Master: 858e41fbf6514476b5e42feda4802e38 (127.4.47.3:41107)
I20260925 07:20:40.837153 4854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:46513
I20260925 07:20:40.847349 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:20:40.855374 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.849244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.4.47.62:36551)
0925 07:20:40.849329 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.849332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.849416 (+ 84us) server_negotiation.cc:407] Connection header received
0925 07:20:40.849530 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.849531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.849542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.849611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:40.850211 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.850698 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.851483 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.851677 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.851777 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.852961 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:40.852962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:40.852962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:40.852969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:40.854182 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.854519 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.854520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.854521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.854580 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:40.854781 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:40.854782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:40.854782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:40.854935 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:40.855043 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.855192 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.855272 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:20:40.857540 4854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:40.859127 4854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:40.866068 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.861628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41107 (local address 127.0.0.1:49598)
0925 07:20:40.861845 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.861890 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.861955 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.862388 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.862390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.862392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:40.862459 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.862461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863874 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.863875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.864966 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.864968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865849 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.865904 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865907 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.865956 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.865995 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":25,"thread_start_us":47,"threads_started":1}
I20260925 07:20:40.866078 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.861794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.4.47.3:41107)
0925 07:20:40.862203 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.862206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.862214 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:40.862259 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.862260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.862274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.862538 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:40.862615 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863763 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865047 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865749 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865843 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865895 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.865967 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.866005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":92,"threads_started":1}
I20260925 07:20:40.866206 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.862006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.4.47.2:38201)
0925 07:20:40.862197 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.862202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.862214 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:20:40.862254 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.862256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.862272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.862360 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:40.862596 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863798 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.864932 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865616 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865712 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865775 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.866024 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.866066 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260925 07:20:40.866351 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.861786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.4.47.1:42225)
0925 07:20:40.862197 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:40.862202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:40.862218 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:20:40.862254 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:40.862256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:40.862268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:40.862366 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:40.862550 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863408 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.864821 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865643 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865715 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865775 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:40.866110 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 07:20:40.866280 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":100,"threads_started":1}
I20260925 07:20:40.866071 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.861683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:42225 (local address 127.0.0.1:41268)
0925 07:20:40.861969 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.861979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.862019 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.862377 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.862381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.862383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:40.862465 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.862468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863514 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.863516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.864739 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.864741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865813 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.865876 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865879 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.865928 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.865979 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260925 07:20:40.866658 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:40.861812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38201 (local address 127.0.0.1:49606)
0925 07:20:40.862008 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:40.862020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:40.862075 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:40.862471 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:40.862472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:40.862474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:40.862516 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.862518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.863915 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.863916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.864851 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:40.864853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:40.865715 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:40.865716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:40.865793 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:40.865807 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:20:40.865876 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:20:40.865933 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260925 07:20:40.866882 5247 tablet_service.cc:1467] Processing CreateTablet for tablet 7eba2bc2a92e449a847e3872422a8354 (DEFAULT_TABLE table=test-table [id=dfbf568373614bd2a76ab395d80ff74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:40.867146 4979 tablet_service.cc:1467] Processing CreateTablet for tablet 7eba2bc2a92e449a847e3872422a8354 (DEFAULT_TABLE table=test-table [id=dfbf568373614bd2a76ab395d80ff74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:40.867285 5247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eba2bc2a92e449a847e3872422a8354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:40.867422 4979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eba2bc2a92e449a847e3872422a8354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:40.867377 5113 tablet_service.cc:1467] Processing CreateTablet for tablet 7eba2bc2a92e449a847e3872422a8354 (DEFAULT_TABLE table=test-table [id=dfbf568373614bd2a76ab395d80ff74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:40.867616 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eba2bc2a92e449a847e3872422a8354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:40.870322 5330 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Bootstrap starting.
I20260925 07:20:40.870565 5332 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Bootstrap starting.
I20260925 07:20:40.871239 5331 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Bootstrap starting.
I20260925 07:20:40.871300 5332 tablet_bootstrap.cc:654] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:40.871300 5330 tablet_bootstrap.cc:654] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:40.871824 5331 tablet_bootstrap.cc:654] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:40.871893 5330 log.cc:824] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:40.871908 5332 log.cc:824] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:40.872473 5331 log.cc:824] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:40.872704 5332 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: No bootstrap required, opened a new log
I20260925 07:20:40.872702 5330 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: No bootstrap required, opened a new log
I20260925 07:20:40.872776 5332 ts_tablet_manager.cc:1397] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:40.872776 5330 ts_tablet_manager.cc:1397] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 07:20:40.873261 5331 tablet_bootstrap.cc:492] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: No bootstrap required, opened a new log
I20260925 07:20:40.873347 5331 ts_tablet_manager.cc:1397] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:40.874673 5331 raft_consensus.cc:348] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.874840 5331 raft_consensus.cc:374] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:40.874886 5331 raft_consensus.cc:729] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858e41fbf6514476b5e42feda4802e38, State: Initialized, Role: FOLLOWER
I20260925 07:20:40.874787 5330 raft_consensus.cc:348] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.874787 5332 raft_consensus.cc:348] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.874965 5332 raft_consensus.cc:374] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:40.874965 5330 raft_consensus.cc:374] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:40.875008 5332 raft_consensus.cc:729] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fcb56a354b49b7b9371b548cd499a5, State: Initialized, Role: FOLLOWER
I20260925 07:20:40.875008 5330 raft_consensus.cc:729] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bee9435cf94a018f49e8b510bc9899, State: Initialized, Role: FOLLOWER
I20260925 07:20:40.875026 5331 consensus_queue.cc:260] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.875161 5332 consensus_queue.cc:260] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.875335 5331 ts_tablet_manager.cc:1428] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:40.875334 5330 consensus_queue.cc:260] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:40.875443 5332 ts_tablet_manager.cc:1428] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:20:40.875463 5313 heartbeater.cc:503] Master 127.4.47.62:36551 was elected leader, sending a full tablet report...
I20260925 07:20:40.875566 5045 heartbeater.cc:503] Master 127.4.47.62:36551 was elected leader, sending a full tablet report...
I20260925 07:20:40.875602 5330 ts_tablet_manager.cc:1428] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 07:20:40.875813 5179 heartbeater.cc:503] Master 127.4.47.62:36551 was elected leader, sending a full tablet report...
W20260925 07:20:40.904742 5180 tablet.cc:2369] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:20:40.985028 5046 tablet.cc:2369] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:41.021639 5336 raft_consensus.cc:482] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:41.021785 5336 raft_consensus.cc:504] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:41.022245 5336 leader_election.cc:288] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83bee9435cf94a018f49e8b510bc9899 (127.4.47.2:38201), 54fcb56a354b49b7b9371b548cd499a5 (127.4.47.1:42225)
I20260925 07:20:41.025503 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.022464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:42225 (local address 127.4.47.3:50603)
0925 07:20:41.022546 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:41.022561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:41.022627 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:41.022815 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:41.022816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:41.022817 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:41.022907 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:41.022910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.023685 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.023686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:41.024625 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:41.024628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.025236 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.025237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:41.025323 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.025327 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:41.025385 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:20:41.025431 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":27}
I20260925 07:20:41.025576 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.022494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:50603 (local address 127.4.47.1:42225)
0925 07:20:41.022561 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:41.022564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:41.022628 (+ 64us) server_negotiation.cc:407] Connection header received
0925 07:20:41.022729 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:41.022730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:41.022745 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:41.022794 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:41.023031 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.023601 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.024740 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.025135 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.025190 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.025199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:41.025430 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 07:20:41.025474 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:20:41.025708 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.022470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38201 (local address 127.4.47.3:36933)
0925 07:20:41.022672 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:41.022683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:41.022754 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:41.022897 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:41.022897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:41.022898 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:41.023009 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:41.023011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.023892 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.023892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:41.024728 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:41.024731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.025433 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.025435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:41.025506 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.025510 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:41.025573 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 07:20:41.025617 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":16,"thread_start_us":111,"threads_started":1}
I20260925 07:20:41.025868 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.022495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:36933 (local address 127.4.47.2:38201)
0925 07:20:41.022561 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:41.022564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:41.022765 (+ 201us) server_negotiation.cc:407] Connection header received
0925 07:20:41.022804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:41.022805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:41.022818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:41.022873 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:41.023263 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.023792 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.024831 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.025302 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.025384 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.025395 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:41.025761 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 07:20:41.025811 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:20:41.025990 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eba2bc2a92e449a847e3872422a8354" candidate_uuid: "858e41fbf6514476b5e42feda4802e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fcb56a354b49b7b9371b548cd499a5" is_pre_election: true
I20260925 07:20:41.026098 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eba2bc2a92e449a847e3872422a8354" candidate_uuid: "858e41fbf6514476b5e42feda4802e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bee9435cf94a018f49e8b510bc9899" is_pre_election: true
I20260925 07:20:41.026181 4999 raft_consensus.cc:2454] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858e41fbf6514476b5e42feda4802e38 in term 0.
I20260925 07:20:41.026228 5133 raft_consensus.cc:2454] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858e41fbf6514476b5e42feda4802e38 in term 0.
I20260925 07:20:41.026440 5203 leader_election.cc:302] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83bee9435cf94a018f49e8b510bc9899, 858e41fbf6514476b5e42feda4802e38; no voters:
I20260925 07:20:41.026592 5336 raft_consensus.cc:2781] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:41.026646 5336 raft_consensus.cc:482] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:41.026669 5336 raft_consensus.cc:3037] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:41.027374 5336 raft_consensus.cc:504] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:41.027527 5336 leader_election.cc:288] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [CANDIDATE]: Term 1 election: Requested vote from peers 83bee9435cf94a018f49e8b510bc9899 (127.4.47.2:38201), 54fcb56a354b49b7b9371b548cd499a5 (127.4.47.1:42225)
I20260925 07:20:41.027732 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eba2bc2a92e449a847e3872422a8354" candidate_uuid: "858e41fbf6514476b5e42feda4802e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83bee9435cf94a018f49e8b510bc9899"
I20260925 07:20:41.027716 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eba2bc2a92e449a847e3872422a8354" candidate_uuid: "858e41fbf6514476b5e42feda4802e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fcb56a354b49b7b9371b548cd499a5"
I20260925 07:20:41.027832 5133 raft_consensus.cc:3037] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:41.027827 4999 raft_consensus.cc:3037] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:41.028554 4999 raft_consensus.cc:2454] T 7eba2bc2a92e449a847e3872422a8354 P 54fcb56a354b49b7b9371b548cd499a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858e41fbf6514476b5e42feda4802e38 in term 1.
I20260925 07:20:41.028577 5133 raft_consensus.cc:2454] T 7eba2bc2a92e449a847e3872422a8354 P 83bee9435cf94a018f49e8b510bc9899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858e41fbf6514476b5e42feda4802e38 in term 1.
I20260925 07:20:41.028743 5201 leader_election.cc:302] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 54fcb56a354b49b7b9371b548cd499a5, 83bee9435cf94a018f49e8b510bc9899, 858e41fbf6514476b5e42feda4802e38; no voters:
I20260925 07:20:41.028851 5336 raft_consensus.cc:2781] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:41.029036 5336 raft_consensus.cc:686] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [term 1 LEADER]: Becoming Leader. State: Replica: 858e41fbf6514476b5e42feda4802e38, State: Running, Role: LEADER
I20260925 07:20:41.029207 5336 consensus_queue.cc:237] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } }
I20260925 07:20:41.030076 4853 catalog_manager.cc:5495] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 858e41fbf6514476b5e42feda4802e38 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "858e41fbf6514476b5e42feda4802e38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858e41fbf6514476b5e42feda4802e38" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83bee9435cf94a018f49e8b510bc9899" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54fcb56a354b49b7b9371b548cd499a5" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42225 } health_report { overall_health: UNKNOWN } } }
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:41.053603 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.045936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.4.47.1:42225)
0925 07:20:41.046033 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:41.046035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:41.046096 (+ 61us) server_negotiation.cc:407] Connection header received
0925 07:20:41.046140 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:41.046141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:41.046153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:41.046207 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:41.046750 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.047325 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.048135 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.048347 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.048415 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.050813 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:41.050824 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:41.050825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:41.050831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:41.052498 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.052835 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.052838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.052839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.052908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.053117 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.053118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.053118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.053257 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:41.053380 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:41.053413 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 07:20:41.053475 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
W20260925 07:20:41.076419 5314 tablet.cc:2369] T 7eba2bc2a92e449a847e3872422a8354 P 858e41fbf6514476b5e42feda4802e38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:41.076783 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.069138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.4.47.1:42225)
0925 07:20:41.069234 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:41.069236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:41.069287 (+ 51us) server_negotiation.cc:407] Connection header received
0925 07:20:41.069333 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:41.069334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:41.069343 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:41.069394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:41.069980 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.070487 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.071350 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.071546 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.071642 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.074030 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:41.074031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:41.074031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:41.074038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:41.075557 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.075921 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.075922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.075922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.075974 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.076196 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.076197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.076197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.076288 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:41.076359 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:41.076652 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 07:20:41.076720 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[4809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[4809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:41.101188 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:41.093000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.4.47.1:42225)
0925 07:20:41.093103 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:41.093105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:41.093251 (+ 146us) server_negotiation.cc:407] Connection header received
0925 07:20:41.093369 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:41.093370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:41.093381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:41.093446 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:41.094060 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.094596 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.095526 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:41.095754 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:41.095846 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:41.098532 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:41.098533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:41.098534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:41.098543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:41.099960 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.100343 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.100345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.100345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.100410 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:41.100656 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:41.100657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:41.100657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:41.100763 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:41.100868 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:41.100979 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 07:20:41.101073 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 07:20:41.101655 4959 tablet_service.cc:2908] Rejecting scan request for tablet 7eba2bc2a92e449a847e3872422a8354: Uninitialized: clean time has not yet been initialized
I20260925 07:20:41.102958 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4916
I20260925 07:20:41.108528 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5050
I20260925 07:20:41.113045 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5184
I20260925 07:20:41.117614 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 4822
2026-09-25T07:20:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2079 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[5354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[5354](info): set up 2 sockets
Sep 25 07:20:41 dist-test-slave-zw07 krb5kdc[5354](info): commencing operation
krb5kdc: starting...
W20260925 07:20:42.151018 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:42.278756 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36005
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:36005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:42.359592 5367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:42.359830 5367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:42.359865 5367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:42.361013 5367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:42.361093 5367 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:20:42.361116 5367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:42.361153 5367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:42.361182 5367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:42.361207 5367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:42.362689 5367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:36005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:42.362963 5367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:42.363222 5367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:42.368119 5375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.368129 5373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.368129 5372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.368485 5367 server_base.cc:1034] running on GCE node
I20260925 07:20:42.368736 5367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:42.369030 5367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:42.370203 5367 hybrid_clock.cc:648] HybridClock initialized: now 1790320842370195 us; error 32 us; skew 500 ppm
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:42.372710 5367 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:42.373129 5367 webserver.cc:460] Webserver started at http://127.4.47.62:36293/ using document root <none> and password file <none>
I20260925 07:20:42.373376 5367 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:42.373442 5367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:42.373588 5367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:42.374679 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "415ead13e3704b38862780760ef2f72a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "d3ad1a8c556a2ac83d9dd707f064f45e"
server_key_iv: "9e80967a50d6d2e5cf53fa342bac7120"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.375039 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "415ead13e3704b38862780760ef2f72a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "d3ad1a8c556a2ac83d9dd707f064f45e"
server_key_iv: "9e80967a50d6d2e5cf53fa342bac7120"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.376437 5367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:42.377197 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.377358 5367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:42.377454 5367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "415ead13e3704b38862780760ef2f72a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "d3ad1a8c556a2ac83d9dd707f064f45e"
server_key_iv: "9e80967a50d6d2e5cf53fa342bac7120"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.377530 5367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:42.390769 5367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:42.392729 5367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:42.392889 5367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:42.397518 5367 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:36005
I20260925 07:20:42.397595 5444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:36005 every 8 connection(s)
I20260925 07:20:42.397919 5367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:42.398550 5445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:42.400894 5445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Bootstrap starting.
I20260925 07:20:42.401486 5445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:42.401773 5445 log.cc:824] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:42.402439 5445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: No bootstrap required, opened a new log
I20260925 07:20:42.403904 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5367
I20260925 07:20:42.403988 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:42.404147 4284 external_mini_cluster.cc:1428] Setting key f98730a67f4000e217b7fd2dda4ede74
I20260925 07:20:42.404816 5445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } }
I20260925 07:20:42.405022 5445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:42.405058 5445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415ead13e3704b38862780760ef2f72a, State: Initialized, Role: FOLLOWER
I20260925 07:20:42.405249 5445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } }
I20260925 07:20:42.405344 5445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:42.405377 5445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:42.405467 5445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:42.406100 5445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } }
I20260925 07:20:42.406212 5445 leader_election.cc:302] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415ead13e3704b38862780760ef2f72a; no voters:
I20260925 07:20:42.406373 5445 leader_election.cc:288] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:42.406442 5450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:42.406608 5450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [term 1 LEADER]: Becoming Leader. State: Replica: 415ead13e3704b38862780760ef2f72a, State: Running, Role: LEADER
I20260925 07:20:42.406626 5445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:42.406709 5450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } }
I20260925 07:20:42.407150 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415ead13e3704b38862780760ef2f72a. Latest consensus state: current_term: 1 leader_uuid: "415ead13e3704b38862780760ef2f72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } } }
I20260925 07:20:42.407243 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:42.407161 5451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415ead13e3704b38862780760ef2f72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415ead13e3704b38862780760ef2f72a" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36005 } } }
I20260925 07:20:42.407310 5451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:42.407788 5459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:42.408290 5459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:42.409852 5459 catalog_manager.cc:1347] Generated new cluster ID: 012f52beb910414a9cbf41d981101825
I20260925 07:20:42.409907 5459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:42.416293 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.404736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.4.47.62:36005)
0925 07:20:42.404934 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.404945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.404981 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:20:42.405110 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.405117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.405178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.405323 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.405981 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.407482 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.408582 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.409143 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.409265 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.412082 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.412086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.412086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.412098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.414566 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.415014 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.415017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.415017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.415160 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.415375 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.415376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.415376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.415673 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.415822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.415941 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.416072 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260925 07:20:42.421953 5459 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:42.422679 5459 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:42.428543 5459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Generated new TSK 0
I20260925 07:20:42.428752 5459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:42.462209 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36005
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:42.562161 5471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:42.562366 5471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:42.562389 5471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:42.563575 5471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:42.563614 5471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:42.563634 5471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:42.563651 5471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:42.565094 5471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:42.565339 5471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:42.565557 5471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:42.570720 5479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.570791 5477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.570788 5476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.570977 5471 server_base.cc:1034] running on GCE node
I20260925 07:20:42.571208 5471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:42.571414 5471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:42.572556 5471 hybrid_clock.cc:648] HybridClock initialized: now 1790320842572536 us; error 37 us; skew 500 ppm
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:42.575048 5471 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:42.575615 5471 webserver.cc:460] Webserver started at http://127.4.47.1:46331/ using document root <none> and password file <none>
I20260925 07:20:42.575860 5471 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:42.575904 5471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:42.576047 5471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:42.577083 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "d5c16f59471341ed97744ed3513bfafa"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "581589d62e5cac15f30218d7ed1efa98"
server_key_iv: "21d4e1bac17fc72b05f4e6fbbbafec4d"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.577437 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "d5c16f59471341ed97744ed3513bfafa"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "581589d62e5cac15f30218d7ed1efa98"
server_key_iv: "21d4e1bac17fc72b05f4e6fbbbafec4d"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.578714 5471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:20:42.579514 5486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.579682 5471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.579766 5471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "d5c16f59471341ed97744ed3513bfafa"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "581589d62e5cac15f30218d7ed1efa98"
server_key_iv: "21d4e1bac17fc72b05f4e6fbbbafec4d"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.579834 5471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:42.604240 5471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:42.606279 5471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:42.606426 5471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:42.607143 5471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:42.607187 5471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.607209 5471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:42.607239 5471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:42.614362 5471 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:45175
I20260925 07:20:42.614434 5601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:45175 every 8 connection(s)
I20260925 07:20:42.614813 5471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:42.616834 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.607561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:51639 (local address 127.4.47.62:36005)
0925 07:20:42.607626 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.607629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.607855 (+ 226us) server_negotiation.cc:407] Connection header received
0925 07:20:42.608394 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.608395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.608407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.608464 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.609354 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.609886 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.611418 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.611657 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.611744 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.614028 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.614029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.614030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.614038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.615726 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.616080 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.616081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.616081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.616131 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.616327 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.616327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.616328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.616429 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.616478 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.616708 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.616771 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:20:42.616983 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.607538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.1:51639)
0925 07:20:42.607738 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.607777 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.608289 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.608547 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.608554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.608903 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.609259 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.609266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.609984 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.609986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.611327 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.611330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.611766 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.611767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.611895 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.612454 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.612463 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.613921 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.615797 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.615801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.615803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.616012 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.616185 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.616185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.616185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.616277 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.616478 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.616482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.616595 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.616682 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.616796 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260925 07:20:42.617525 5602 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36005
I20260925 07:20:42.617600 5602 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:42.617786 5602 heartbeater.cc:511] Master 127.4.47.62:36005 requested a full tablet report, sending...
I20260925 07:20:42.618089 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5471
I20260925 07:20:42.618178 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:42.618324 4284 external_mini_cluster.cc:1428] Setting key 723fa3fc0476863fd92832fdc734d0b2
I20260925 07:20:42.618674 5398 ts_manager.cc:194] Registered new tserver with Master: d5c16f59471341ed97744ed3513bfafa (127.4.47.1:45175)
I20260925 07:20:42.619391 5398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:51639
I20260925 07:20:42.625479 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.619605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52965 (local address 127.4.47.62:36005)
0925 07:20:42.619818 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.619820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.619828 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:42.619848 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.619849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.619859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.619919 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.620294 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.620762 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.621658 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.621833 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.621919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.622970 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.622971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.622971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.622978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.624393 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.624743 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.624744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.624744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.624802 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.624987 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.624988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.624988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.625096 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.625170 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.625311 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.625388 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 07:20:42.625494 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.618571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.1:52965)
0925 07:20:42.618722 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.618734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.618787 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.619936 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.619937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.620136 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.620210 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.620213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.620854 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.620855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.621581 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.621583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.621917 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.621918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.622004 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.622512 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.622517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.622888 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.624490 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.624491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.624492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.624662 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.624863 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.624864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.624865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.624921 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.625161 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.625162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.625206 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.625288 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.625387 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:42.646027 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36005
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:42.732751 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:42.732995 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:42.733033 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:42.734213 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:42.734275 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:42.734308 5611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:42.734345 5611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:42.735790 5611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:42.736078 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:42.736305 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:42.741278 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.741298 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.741333 5611 server_base.cc:1034] running on GCE node
W20260925 07:20:42.741328 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.741709 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:42.741916 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:42.743132 5611 hybrid_clock.cc:648] HybridClock initialized: now 1790320842743086 us; error 68 us; skew 500 ppm
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:42.745867 5611 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:42.746376 5611 webserver.cc:460] Webserver started at http://127.4.47.2:39187/ using document root <none> and password file <none>
I20260925 07:20:42.746603 5611 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:42.746685 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:42.746829 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:42.748004 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "4159436df17261fd05834d31390fbbcd"
server_key_iv: "2a0d12c38090ec0224cb71446faee879"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.748371 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "4159436df17261fd05834d31390fbbcd"
server_key_iv: "2a0d12c38090ec0224cb71446faee879"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.749733 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:20:42.750466 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.750641 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:42.750744 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "4159436df17261fd05834d31390fbbcd"
server_key_iv: "2a0d12c38090ec0224cb71446faee879"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.750824 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:42.761917 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:42.763967 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:42.764132 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:42.764721 5611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:42.764775 5611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.764817 5611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:42.764843 5611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:42.772101 5611 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:38301
I20260925 07:20:42.772469 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:42.772493 5741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:38301 every 8 connection(s)
I20260925 07:20:42.774417 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.765115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:40919 (local address 127.4.47.62:36005)
0925 07:20:42.765228 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.765230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.765405 (+ 175us) server_negotiation.cc:407] Connection header received
0925 07:20:42.765892 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.765893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.765905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.765989 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.766769 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.767402 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.768396 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.768664 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.768724 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.771680 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.771681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.771682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.771688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.773222 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.773565 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.773566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.773567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.773645 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.773887 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.773888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.773888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.773993 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.774047 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.774292 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.774354 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 07:20:42.774654 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.765115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.2:40919)
0925 07:20:42.765273 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.765314 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.765799 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.766052 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.766056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.766476 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.766675 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.766679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.767489 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.767490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.768298 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.768300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.768787 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.768788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.768887 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.769382 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.769395 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.771592 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.773291 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.773295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.773297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.773503 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.773718 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.773718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.773719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.773832 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.774048 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.774052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.774179 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.774264 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.774384 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 07:20:42.775127 5742 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36005
I20260925 07:20:42.775215 5742 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:42.775399 5742 heartbeater.cc:511] Master 127.4.47.62:36005 requested a full tablet report, sending...
I20260925 07:20:42.775758 5399 ts_manager.cc:194] Registered new tserver with Master: a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301)
I20260925 07:20:42.776186 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:40919
I20260925 07:20:42.781006 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5611
I20260925 07:20:42.781111 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:42.781323 4284 external_mini_cluster.cc:1428] Setting key 6b736947db584bd72fa9671b132591e7
I20260925 07:20:42.781597 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.776187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.2:59935)
0925 07:20:42.776300 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.776311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.776373 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.776494 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.776494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.776712 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.776768 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.776770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.777369 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.777370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.777872 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.777874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.778212 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.778213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.778269 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.778650 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.778654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.778917 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.780507 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.780508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.780509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.780687 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.780846 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.780847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.780847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.780914 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.781269 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.781269 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.781306 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.781376 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.781488 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260925 07:20:42.781603 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.776245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:59935 (local address 127.4.47.62:36005)
0925 07:20:42.776316 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.776318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.776357 (+ 39us) server_negotiation.cc:407] Connection header received
0925 07:20:42.776421 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.776422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.776431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.776491 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.776845 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.777286 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.777944 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.778116 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.778185 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.778985 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.778986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.778986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.778992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.780426 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.780744 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.780745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.780745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.780793 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.780974 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.780975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.780975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.781161 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.781235 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.781457 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.781523 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:42.806658 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36005
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:42.894778 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:42.894989 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:42.895013 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:42.896224 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:42.896267 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:42.896293 5751 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:42.896334 5751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:42.897749 5751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:42.898008 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:42.898244 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:42.903725 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.903754 5751 server_base.cc:1034] running on GCE node
W20260925 07:20:42.903873 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:42.903946 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:42.904246 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:42.904484 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:42.905639 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790320842905624 us; error 31 us; skew 500 ppm
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:42.908777 5751 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:42.909267 5751 webserver.cc:460] Webserver started at http://127.4.47.3:38107/ using document root <none> and password file <none>
I20260925 07:20:42.909494 5751 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:42.909540 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:42.909680 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:42.910755 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "7fa49e70cf8042d1828acdb94aa39854"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "a8ada2882d17c09fc71080acfb03ac03"
server_key_iv: "7086ea738e25941c258cd20def0b0a08"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.911165 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "7fa49e70cf8042d1828acdb94aa39854"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "a8ada2882d17c09fc71080acfb03ac03"
server_key_iv: "7086ea738e25941c258cd20def0b0a08"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.912518 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:42.913302 5766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.913488 5751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:42.913586 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "7fa49e70cf8042d1828acdb94aa39854"
format_stamp: "Formatted at 2026-09-25 07:20:42 on dist-test-slave-zw07"
server_key: "a8ada2882d17c09fc71080acfb03ac03"
server_key_iv: "7086ea738e25941c258cd20def0b0a08"
server_key_version: "encryptionkey@0"
I20260925 07:20:42.913681 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:42.929152 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:42.932001 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:42.932163 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:42.932844 5751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:42.932899 5751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:42.932935 5751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:42.932963 5751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:42.941039 5751 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:41237
I20260925 07:20:42.941133 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:41237 every 8 connection(s)
I20260925 07:20:42.941486 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:20:42.942526 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5751
I20260925 07:20:42.942611 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:42.942797 4284 external_mini_cluster.cc:1428] Setting key 828788a2073deab5ed3aaa86d1298629
I20260925 07:20:42.944041 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.933231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:58631 (local address 127.4.47.62:36005)
0925 07:20:42.933280 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.933282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.933512 (+ 230us) server_negotiation.cc:407] Connection header received
0925 07:20:42.934068 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.934069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.934081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.934134 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.934942 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.935772 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.937369 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.937695 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.937762 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.940666 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.940667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.940667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.940676 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.942287 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.942834 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.942835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.942836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.942910 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.943249 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.943250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.943250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.943409 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.943480 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.943822 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.943923 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:42.944099 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.933238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.3:58631)
0925 07:20:42.933392 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.933432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.933955 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.934212 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.934215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.934510 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.934822 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.934829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.935876 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.935877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.937260 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.937262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.937776 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.937777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.937933 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.938671 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.938691 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.940552 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.942374 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.942381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.942384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.942740 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.943010 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.943011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.943012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.943171 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.943470 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.943475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.943649 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.943785 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.943919 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260925 07:20:42.944844 5882 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36005
I20260925 07:20:42.944944 5882 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:42.945123 5882 heartbeater.cc:511] Master 127.4.47.62:36005 requested a full tablet report, sending...
I20260925 07:20:42.945492 5399 ts_manager.cc:194] Registered new tserver with Master: 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237)
I20260925 07:20:42.945885 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:58631
I20260925 07:20:42.952098 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.945999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:45827 (local address 127.4.47.62:36005)
0925 07:20:42.946048 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.946051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.946116 (+ 65us) server_negotiation.cc:407] Connection header received
0925 07:20:42.946188 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.946189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.946199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.946258 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.946714 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.947249 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.948385 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.948562 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.948609 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.949637 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.949638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.949639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.949644 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.951019 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.951428 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.951429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.951429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.951479 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.951667 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.951668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.951668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.951762 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.951812 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.951973 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.952033 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:20:42.952116 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.945930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.4.47.3:45827)
0925 07:20:42.946060 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.946072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.946130 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.946305 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.946307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.946534 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:42.946635 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.946637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.947333 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.947334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.948289 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.948291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.948625 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.948632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.948712 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.949142 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:42.949148 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:42.949557 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:42.951130 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.951132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.951132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.951359 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.951537 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:42.951538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:42.951539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:42.951601 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:42.951818 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:42.951819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:42.951868 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.951942 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.952041 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260925 07:20:42.954501 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:20:42 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:42.977068 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.969977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.4.47.62:36005)
0925 07:20:42.970049 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.970051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.970060 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:20:42.970135 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.970136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.970146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.970209 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:42.970754 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.971394 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.972197 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.972387 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.972445 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.974698 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:42.974700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:42.974700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:42.974713 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:42.976099 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.976439 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.976440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.976440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.976494 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:42.976652 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:42.976653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:42.976653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:42.976768 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:42.976842 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.976924 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.976981 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:20:42.978592 5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:42.980239 5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:42.986982 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.982801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.4.47.2:38301)
0925 07:20:42.983057 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.983095 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.983311 (+ 216us) server_negotiation.cc:407] Connection header received
0925 07:20:42.983362 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.983364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.983380 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.983453 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:42.983706 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984439 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986014 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986500 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.986666 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.986822 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.986877 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260925 07:20:42.986984 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.982914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41237 (local address 127.0.0.1:33404)
0925 07:20:42.983219 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.983227 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.983270 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.983526 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.983527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.983529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:42.983626 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.983635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984878 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.984879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.985778 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.985780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986700 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.986763 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.986768 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.986799 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.986881 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":49,"thread_start_us":60,"threads_started":1}
I20260925 07:20:42.987134 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.982704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38301 (local address 127.0.0.1:48946)
0925 07:20:42.983141 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.983151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.983249 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.983518 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.983518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.983520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:42.983625 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.983628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984525 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.984526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.985932 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.985934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986601 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.986694 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.986708 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.986804 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.987030 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":73,"thread_start_us":91,"threads_started":1}
I20260925 07:20:42.986995 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.982699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45175 (local address 127.0.0.1:60030)
0925 07:20:42.982848 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:42.982884 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:42.982983 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:42.983384 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:42.983386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:42.983388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:42.983461 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.983462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984841 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.984842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.985902 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:42.985904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986733 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:42.986785 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.986788 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:42.986817 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 07:20:42.986907 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":43,"thread_start_us":52,"threads_started":1}
I20260925 07:20:42.987418 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.983015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.4.47.1:45175)
0925 07:20:42.983202 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.983207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.983218 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:20:42.983257 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.983259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.983274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.983370 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:42.983605 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984753 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.985984 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986648 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.986861 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.986922 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.986953 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.986990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260925 07:20:42.987656 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:42.983032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.4.47.3:41237)
0925 07:20:42.983383 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:42.983387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:42.983396 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:20:42.983432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:42.983434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:42.983448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:42.983520 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:42.983720 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.984777 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.985866 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:42.986593 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:42.987428 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:42.987500 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:42.987527 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 07:20:42.987569 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260925 07:20:42.987854 5676 tablet_service.cc:1467] Processing CreateTablet for tablet ee8fefb3488a43948168a524289358df (DEFAULT_TABLE table=test-table [id=6a93a7874ee440d5b75f702156bcab3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:42.988025 5536 tablet_service.cc:1467] Processing CreateTablet for tablet ee8fefb3488a43948168a524289358df (DEFAULT_TABLE table=test-table [id=6a93a7874ee440d5b75f702156bcab3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:42.988255 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8fefb3488a43948168a524289358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:42.988304 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8fefb3488a43948168a524289358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:42.992025 5906 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Bootstrap starting.
I20260925 07:20:42.992421 5816 tablet_service.cc:1467] Processing CreateTablet for tablet ee8fefb3488a43948168a524289358df (DEFAULT_TABLE table=test-table [id=6a93a7874ee440d5b75f702156bcab3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:42.992653 5906 tablet_bootstrap.cc:654] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:42.992740 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8fefb3488a43948168a524289358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:42.992976 5906 log.cc:824] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:42.993291 5905 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Bootstrap starting.
I20260925 07:20:42.993744 5906 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: No bootstrap required, opened a new log
I20260925 07:20:42.993815 5906 ts_tablet_manager.cc:1397] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:42.993820 5905 tablet_bootstrap.cc:654] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:42.994138 5905 log.cc:824] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:42.994800 5905 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: No bootstrap required, opened a new log
I20260925 07:20:42.994884 5905 ts_tablet_manager.cc:1397] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:42.995919 5906 raft_consensus.cc:348] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:42.996101 5906 raft_consensus.cc:374] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:42.996146 5906 raft_consensus.cc:729] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f373c3e8094b5d8ffbc73c2620e88a, State: Initialized, Role: FOLLOWER
I20260925 07:20:42.996227 5909 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Bootstrap starting.
I20260925 07:20:42.996284 5906 consensus_queue.cc:260] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:42.996585 5906 ts_tablet_manager.cc:1428] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:20:42.996603 5742 heartbeater.cc:503] Master 127.4.47.62:36005 was elected leader, sending a full tablet report...
I20260925 07:20:42.996529 5905 raft_consensus.cc:348] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:42.996690 5905 raft_consensus.cc:374] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:42.996729 5905 raft_consensus.cc:729] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c16f59471341ed97744ed3513bfafa, State: Initialized, Role: FOLLOWER
I20260925 07:20:42.996835 5905 consensus_queue.cc:260] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:42.997151 5905 ts_tablet_manager.cc:1428] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:20:42.997164 5602 heartbeater.cc:503] Master 127.4.47.62:36005 was elected leader, sending a full tablet report...
I20260925 07:20:42.997592 5909 tablet_bootstrap.cc:654] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:42.997934 5909 log.cc:824] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:42.998732 5909 tablet_bootstrap.cc:492] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: No bootstrap required, opened a new log
I20260925 07:20:42.998785 5909 ts_tablet_manager.cc:1397] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:20:43.000146 5909 raft_consensus.cc:348] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.000263 5909 raft_consensus.cc:374] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.000284 5909 raft_consensus.cc:729] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa49e70cf8042d1828acdb94aa39854, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.000384 5909 consensus_queue.cc:260] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.000599 5909 ts_tablet_manager.cc:1428] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:20:43.000663 5882 heartbeater.cc:503] Master 127.4.47.62:36005 was elected leader, sending a full tablet report...
W20260925 07:20:43.023231 5743 tablet.cc:2369] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:43.051260 5911 raft_consensus.cc:482] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:43.051391 5911 raft_consensus.cc:504] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.051697 5911 leader_election.cc:288] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301), 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237)
I20260925 07:20:43.055285 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.052156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:57537 (local address 127.4.47.3:41237)
0925 07:20:43.052222 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.052224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.052274 (+ 50us) server_negotiation.cc:407] Connection header received
0925 07:20:43.052341 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.052342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.052352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.052423 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:43.052585 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.053071 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054329 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.054895 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054972 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.054982 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.055171 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.055216 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:20:43.055310 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.051995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:38913 (local address 127.4.47.2:38301)
0925 07:20:43.052062 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.052064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.052076 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:20:43.052108 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.052110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.052124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.052204 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:43.052360 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.052877 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054329 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.054901 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054972 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.054982 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.055212 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.055249 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:20:43.055302 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.051912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38301 (local address 127.4.47.1:38913)
0925 07:20:43.051966 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.051979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.052024 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.052210 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.052211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.052214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:43.052285 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.052287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.052977 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.052978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.054223 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.054226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.054983 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.055092 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.055096 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.055209 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.055241 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:20:43.055310 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.052047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41237 (local address 127.4.47.1:57537)
0925 07:20:43.052225 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.052234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.052287 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.052425 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.052426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.052427 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:43.052506 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.052507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.053160 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.053161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.054247 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.054249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.054983 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.054984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.055098 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.055101 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.055156 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.055206 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":13,"thread_start_us":120,"threads_started":1}
I20260925 07:20:43.055719 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8fefb3488a43948168a524289358df" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" is_pre_election: true
I20260925 07:20:43.055886 5695 raft_consensus.cc:2454] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 0.
I20260925 07:20:43.055938 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8fefb3488a43948168a524289358df" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa49e70cf8042d1828acdb94aa39854" is_pre_election: true
I20260925 07:20:43.056119 5836 raft_consensus.cc:2454] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 0.
I20260925 07:20:43.056125 5490 leader_election.cc:302] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f373c3e8094b5d8ffbc73c2620e88a, d5c16f59471341ed97744ed3513bfafa; no voters:
I20260925 07:20:43.056255 5911 raft_consensus.cc:2781] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:43.056315 5911 raft_consensus.cc:482] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:43.056341 5911 raft_consensus.cc:3037] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.057284 5911 raft_consensus.cc:504] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.057420 5911 leader_election.cc:288] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 election: Requested vote from peers a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301), 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237)
I20260925 07:20:43.057586 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8fefb3488a43948168a524289358df" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
I20260925 07:20:43.057596 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8fefb3488a43948168a524289358df" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa49e70cf8042d1828acdb94aa39854"
I20260925 07:20:43.057663 5836 raft_consensus.cc:3037] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.057669 5695 raft_consensus.cc:3037] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.058350 5836 raft_consensus.cc:2454] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 1.
I20260925 07:20:43.058375 5695 raft_consensus.cc:2454] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 1.
I20260925 07:20:43.058508 5490 leader_election.cc:302] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa49e70cf8042d1828acdb94aa39854, d5c16f59471341ed97744ed3513bfafa; no voters:
I20260925 07:20:43.058634 5911 raft_consensus.cc:2781] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:43.058804 5911 raft_consensus.cc:686] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 1 LEADER]: Becoming Leader. State: Replica: d5c16f59471341ed97744ed3513bfafa, State: Running, Role: LEADER
I20260925 07:20:43.058908 5911 consensus_queue.cc:237] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.059840 5399 catalog_manager.cc:5495] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c16f59471341ed97744ed3513bfafa (127.4.47.1). New cstate: current_term: 1 leader_uuid: "d5c16f59471341ed97744ed3513bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20260925 07:20:43.080428 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.077706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.4.47.1:45175)
0925 07:20:43.077775 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.077777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.077788 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:20:43.077902 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.077903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.077930 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.078000 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:43.078204 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.078682 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.079502 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.079705 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.079785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.080060 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.080163 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.080253 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.080295 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.080316 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:20:43.082455 5695 raft_consensus.cc:1261] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Refusing update from remote peer d5c16f59471341ed97744ed3513bfafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.082455 5836 raft_consensus.cc:1261] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Refusing update from remote peer d5c16f59471341ed97744ed3513bfafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.082861 5911 consensus_queue.cc:1035] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.083016 5915 consensus_queue.cc:1035] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.089344 5399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:43.089453 5399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:43.090288 5399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Sending DeleteTablet for 3 replicas of tablet ee8fefb3488a43948168a524289358df
I20260925 07:20:43.090615 5536 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8fefb3488a43948168a524289358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60030
I20260925 07:20:43.090651 5676 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8fefb3488a43948168a524289358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260925 07:20:43.090925 5816 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8fefb3488a43948168a524289358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:33404
I20260925 07:20:43.090920 5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:43.091058 5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:43.091161 5929 tablet_replica.cc:330] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: stopping tablet replica
I20260925 07:20:43.091265 5929 raft_consensus.cc:2229] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:43.091383 5929 raft_consensus.cc:2258] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.091655 5928 tablet_replica.cc:330] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: stopping tablet replica
I20260925 07:20:43.091758 5928 raft_consensus.cc:2229] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:43.092128 5928 raft_consensus.cc:2258] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.092389 5816 tablet_service.cc:1467] Processing CreateTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 (DEFAULT_TABLE table=test-table [id=3f921b8fbb8040a196ab85e06dd9b587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:43.092507 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e33f7b664cb4c0c94df1af3542b8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.092495 5536 tablet_service.cc:1467] Processing CreateTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 (DEFAULT_TABLE table=test-table [id=3f921b8fbb8040a196ab85e06dd9b587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:43.092617 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e33f7b664cb4c0c94df1af3542b8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.092639 5929 ts_tablet_manager.cc:1905] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.093881 5909 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Bootstrap starting.
I20260925 07:20:43.093912 5905 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Bootstrap starting.
I20260925 07:20:43.093981 5676 tablet_service.cc:1467] Processing CreateTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 (DEFAULT_TABLE table=test-table [id=3f921b8fbb8040a196ab85e06dd9b587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:43.094084 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e33f7b664cb4c0c94df1af3542b8b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.094296 5905 tablet_bootstrap.cc:654] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.094293 5909 tablet_bootstrap.cc:654] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.094777 5927 tablet_replica.cc:330] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: stopping tablet replica
I20260925 07:20:43.094870 5927 raft_consensus.cc:2229] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:43.094983 5927 raft_consensus.cc:2258] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.095083 5928 ts_tablet_manager.cc:1905] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.095408 5927 ts_tablet_manager.cc:1905] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.095474 5906 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Bootstrap starting.
I20260925 07:20:43.095840 5929 ts_tablet_manager.cc:1918] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:43.095884 5906 tablet_bootstrap.cc:654] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.095908 5929 log.cc:1196] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/ee8fefb3488a43948168a524289358df
I20260925 07:20:43.096179 5929 ts_tablet_manager.cc:1939] T ee8fefb3488a43948168a524289358df P 7fa49e70cf8042d1828acdb94aa39854: Deleting consensus metadata
I20260925 07:20:43.096562 5386 catalog_manager.cc:4841] TS 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237): tablet ee8fefb3488a43948168a524289358df (table test-table [id=6a93a7874ee440d5b75f702156bcab3b]) successfully deleted
I20260925 07:20:43.096627 5928 ts_tablet_manager.cc:1918] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:43.096706 5928 log.cc:1196] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/ee8fefb3488a43948168a524289358df
I20260925 07:20:43.096952 5905 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: No bootstrap required, opened a new log
I20260925 07:20:43.096976 5928 ts_tablet_manager.cc:1939] T ee8fefb3488a43948168a524289358df P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting consensus metadata
I20260925 07:20:43.097004 5905 ts_tablet_manager.cc:1397] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:20:43.097061 5906 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: No bootstrap required, opened a new log
I20260925 07:20:43.097100 5906 ts_tablet_manager.cc:1397] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:43.097164 5905 raft_consensus.cc:348] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097249 5905 raft_consensus.cc:374] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.097270 5905 raft_consensus.cc:729] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c16f59471341ed97744ed3513bfafa, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.097244 5906 raft_consensus.cc:348] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097321 5906 raft_consensus.cc:374] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.097347 5906 raft_consensus.cc:729] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f373c3e8094b5d8ffbc73c2620e88a, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.097322 5905 consensus_queue.cc:260] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097371 5386 catalog_manager.cc:4841] TS a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301): tablet ee8fefb3488a43948168a524289358df (table test-table [id=6a93a7874ee440d5b75f702156bcab3b]) successfully deleted
I20260925 07:20:43.097388 5906 consensus_queue.cc:260] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097419 5909 tablet_bootstrap.cc:492] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: No bootstrap required, opened a new log
I20260925 07:20:43.097417 5905 ts_tablet_manager.cc:1428] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:43.097456 5909 ts_tablet_manager.cc:1397] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 07:20:43.097468 5906 ts_tablet_manager.cc:1428] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:43.097605 5909 raft_consensus.cc:348] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097694 5909 raft_consensus.cc:374] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.097723 5909 raft_consensus.cc:729] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa49e70cf8042d1828acdb94aa39854, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.097764 5909 consensus_queue.cc:260] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.097846 5909 ts_tablet_manager.cc:1428] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:43.097272 5927 ts_tablet_manager.cc:1918] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:43.097996 5927 log.cc:1196] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/ee8fefb3488a43948168a524289358df
I20260925 07:20:43.098259 5927 ts_tablet_manager.cc:1939] T ee8fefb3488a43948168a524289358df P d5c16f59471341ed97744ed3513bfafa: Deleting consensus metadata
I20260925 07:20:43.098608 5384 catalog_manager.cc:4841] TS d5c16f59471341ed97744ed3513bfafa (127.4.47.1:45175): tablet ee8fefb3488a43948168a524289358df (table test-table [id=6a93a7874ee440d5b75f702156bcab3b]) successfully deleted
W20260925 07:20:43.115724 5603 tablet.cc:2369] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:43.175696 5911 raft_consensus.cc:482] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:43.175791 5911 raft_consensus.cc:504] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.175947 5911 leader_election.cc:288] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301), 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237)
I20260925 07:20:43.176203 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e33f7b664cb4c0c94df1af3542b8b33" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" is_pre_election: true
I20260925 07:20:43.176218 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e33f7b664cb4c0c94df1af3542b8b33" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa49e70cf8042d1828acdb94aa39854" is_pre_election: true
I20260925 07:20:43.176302 5836 raft_consensus.cc:2454] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 0.
I20260925 07:20:43.176302 5695 raft_consensus.cc:2454] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 0.
I20260925 07:20:43.176455 5490 leader_election.cc:302] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa49e70cf8042d1828acdb94aa39854, d5c16f59471341ed97744ed3513bfafa; no voters:
I20260925 07:20:43.176555 5911 raft_consensus.cc:2781] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:43.176596 5911 raft_consensus.cc:482] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:43.176617 5911 raft_consensus.cc:3037] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.177289 5911 raft_consensus.cc:504] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.177412 5911 leader_election.cc:288] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 election: Requested vote from peers a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301), 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237)
I20260925 07:20:43.177565 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e33f7b664cb4c0c94df1af3542b8b33" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
I20260925 07:20:43.177652 5695 raft_consensus.cc:3037] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.177599 5836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e33f7b664cb4c0c94df1af3542b8b33" candidate_uuid: "d5c16f59471341ed97744ed3513bfafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa49e70cf8042d1828acdb94aa39854"
I20260925 07:20:43.177695 5836 raft_consensus.cc:3037] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.178337 5695 raft_consensus.cc:2454] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 1.
I20260925 07:20:43.178506 5490 leader_election.cc:302] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f373c3e8094b5d8ffbc73c2620e88a, d5c16f59471341ed97744ed3513bfafa; no voters:
I20260925 07:20:43.178517 5836 raft_consensus.cc:2454] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c16f59471341ed97744ed3513bfafa in term 1.
I20260925 07:20:43.178602 5911 raft_consensus.cc:2781] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:43.178655 5911 raft_consensus.cc:686] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 1 LEADER]: Becoming Leader. State: Replica: d5c16f59471341ed97744ed3513bfafa, State: Running, Role: LEADER
I20260925 07:20:43.178711 5911 consensus_queue.cc:237] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.179378 5399 catalog_manager.cc:5495] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c16f59471341ed97744ed3513bfafa (127.4.47.1). New cstate: current_term: 1 leader_uuid: "d5c16f59471341ed97744ed3513bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } health_report { overall_health: UNKNOWN } } }
W20260925 07:20:43.192032 5883 tablet.cc:2369] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 07:20:43 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:43.222656 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.215832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36005 (local address 127.0.0.1:37834)
0925 07:20:43.215997 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.216008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.216079 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.216239 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.216240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.216539 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:43.216587 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.216589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.217286 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.217286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.217941 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.217943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.218320 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.218321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.218367 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.218864 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:43.218869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:43.220332 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:43.221756 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:43.221762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:43.221764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:43.221957 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:43.222139 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:43.222140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:43.222140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:43.222195 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:43.222389 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:43.222398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:43.222439 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.222490 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.222579 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260925 07:20:43.222757 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.215878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.4.47.62:36005)
0925 07:20:43.215946 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.215948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.216074 (+ 126us) server_negotiation.cc:407] Connection header received
0925 07:20:43.216160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.216161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.216171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.216224 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:43.216652 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.217210 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.218022 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.218243 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.218295 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.220412 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:43.220414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:43.220414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:43.220420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:43.221685 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.222019 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.222020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.222020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.222067 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.222244 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.222245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.222245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.222340 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:43.222390 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.222515 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.222634 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":37}
I20260925 07:20:43.224653 5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:37834:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:20:43.224888 5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:43.226280 5536 tablet_service.cc:1467] Processing CreateTablet for tablet 28af3f6e2ce4481d92bd48d145721b07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4af2652a4a214626adb3e8921deb711d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:43.226289 5676 tablet_service.cc:1467] Processing CreateTablet for tablet 28af3f6e2ce4481d92bd48d145721b07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4af2652a4a214626adb3e8921deb711d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:43.226325 5816 tablet_service.cc:1467] Processing CreateTablet for tablet 28af3f6e2ce4481d92bd48d145721b07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4af2652a4a214626adb3e8921deb711d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:43.226441 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28af3f6e2ce4481d92bd48d145721b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.226442 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28af3f6e2ce4481d92bd48d145721b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.226473 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28af3f6e2ce4481d92bd48d145721b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:43.227964 5906 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: Bootstrap starting.
I20260925 07:20:43.227964 5905 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: Bootstrap starting.
I20260925 07:20:43.227977 5909 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: Bootstrap starting.
I20260925 07:20:43.228374 5909 tablet_bootstrap.cc:654] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.228368 5905 tablet_bootstrap.cc:654] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.228368 5906 tablet_bootstrap.cc:654] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:43.228968 5905 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: No bootstrap required, opened a new log
I20260925 07:20:43.229034 5905 ts_tablet_manager.cc:1397] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:43.228995 5906 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: No bootstrap required, opened a new log
I20260925 07:20:43.229174 5909 tablet_bootstrap.cc:492] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: No bootstrap required, opened a new log
I20260925 07:20:43.229179 5906 ts_tablet_manager.cc:1397] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:43.229144 5905 raft_consensus.cc:348] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229214 5905 raft_consensus.cc:374] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.229210 5909 ts_tablet_manager.cc:1397] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:43.229236 5905 raft_consensus.cc:729] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c16f59471341ed97744ed3513bfafa, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.229296 5905 consensus_queue.cc:260] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229382 5905 ts_tablet_manager.cc:1428] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:43.229337 5906 raft_consensus.cc:348] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229346 5909 raft_consensus.cc:348] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229417 5906 raft_consensus.cc:374] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.229426 5909 raft_consensus.cc:374] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:43.229439 5906 raft_consensus.cc:729] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f373c3e8094b5d8ffbc73c2620e88a, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.229450 5909 raft_consensus.cc:729] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa49e70cf8042d1828acdb94aa39854, State: Initialized, Role: FOLLOWER
I20260925 07:20:43.229409 5915 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.229477 5906 consensus_queue.cc:260] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229491 5915 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.229490 5909 consensus_queue.cc:260] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.229568 5906 ts_tablet_manager.cc:1428] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:43.229571 5909 ts_tablet_manager.cc:1428] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:43.229626 5910 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.229676 5913 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.229723 5910 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.229748 5913 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.310385 5913 raft_consensus.cc:482] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:43.310491 5913 raft_consensus.cc:504] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.310879 5913 leader_election.cc:288] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c16f59471341ed97744ed3513bfafa (127.4.47.1:45175), a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301)
I20260925 07:20:43.314656 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.311048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38301 (local address 127.4.47.3:50181)
0925 07:20:43.311219 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.311230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.311288 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.311468 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.311469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.311471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:43.311556 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.311559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.312499 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.312500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.313834 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.313837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.314454 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.314455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.314510 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.314514 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.314555 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.314590 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260925 07:20:43.314697 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.311135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:50181 (local address 127.4.47.2:38301)
0925 07:20:43.311206 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.311209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.311275 (+ 66us) server_negotiation.cc:407] Connection header received
0925 07:20:43.311362 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.311364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.311378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.311452 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:43.311628 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.312406 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.313921 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.314380 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.314450 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.314462 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.314579 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.314617 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:20:43.314900 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28af3f6e2ce4481d92bd48d145721b07" candidate_uuid: "7fa49e70cf8042d1828acdb94aa39854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" is_pre_election: true
I20260925 07:20:43.315006 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.311165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:41733 (local address 127.4.47.1:45175)
0925 07:20:43.311216 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.311218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.311417 (+ 199us) server_negotiation.cc:407] Connection header received
0925 07:20:43.311479 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.311480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.311489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.311545 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:43.311703 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.312395 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.314123 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.314654 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.314720 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.314730 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.314901 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.314944 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:43.315011 5695 raft_consensus.cc:2454] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa49e70cf8042d1828acdb94aa39854 in term 0.
I20260925 07:20:43.315295 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.311065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45175 (local address 127.4.47.3:41733)
0925 07:20:43.311337 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.311351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.311406 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.311551 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.311551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.311552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:43.311623 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.311624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.312499 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.312500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.314050 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.314052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.314741 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.314742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.314814 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.314818 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.315170 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.315219 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":112,"thread_start_us":99,"threads_started":1}
I20260925 07:20:43.315284 5770 leader_election.cc:302] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa49e70cf8042d1828acdb94aa39854, a9f373c3e8094b5d8ffbc73c2620e88a; no voters:
I20260925 07:20:43.315431 5913 raft_consensus.cc:2781] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:43.315471 5913 raft_consensus.cc:482] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:43.315492 5913 raft_consensus.cc:3037] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.315565 5556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28af3f6e2ce4481d92bd48d145721b07" candidate_uuid: "7fa49e70cf8042d1828acdb94aa39854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c16f59471341ed97744ed3513bfafa" is_pre_election: true
I20260925 07:20:43.315711 5556 raft_consensus.cc:2454] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa49e70cf8042d1828acdb94aa39854 in term 0.
I20260925 07:20:43.316282 5913 raft_consensus.cc:504] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.316426 5913 leader_election.cc:288] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [CANDIDATE]: Term 1 election: Requested vote from peers d5c16f59471341ed97744ed3513bfafa (127.4.47.1:45175), a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301)
I20260925 07:20:43.316571 5556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28af3f6e2ce4481d92bd48d145721b07" candidate_uuid: "7fa49e70cf8042d1828acdb94aa39854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c16f59471341ed97744ed3513bfafa"
I20260925 07:20:43.316608 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28af3f6e2ce4481d92bd48d145721b07" candidate_uuid: "7fa49e70cf8042d1828acdb94aa39854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a"
I20260925 07:20:43.316671 5556 raft_consensus.cc:3037] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.316689 5695 raft_consensus.cc:3037] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:43.317375 5556 raft_consensus.cc:2454] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa49e70cf8042d1828acdb94aa39854 in term 1.
I20260925 07:20:43.317380 5695 raft_consensus.cc:2454] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa49e70cf8042d1828acdb94aa39854 in term 1.
I20260925 07:20:43.317579 5770 leader_election.cc:302] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa49e70cf8042d1828acdb94aa39854, a9f373c3e8094b5d8ffbc73c2620e88a; no voters:
I20260925 07:20:43.317698 5913 raft_consensus.cc:2781] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:43.317847 5913 raft_consensus.cc:686] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa49e70cf8042d1828acdb94aa39854, State: Running, Role: LEADER
I20260925 07:20:43.317926 5913 consensus_queue.cc:237] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } }
I20260925 07:20:43.318334 5944 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: TxnStatusTablet state changed. Reason: New leader 7fa49e70cf8042d1828acdb94aa39854. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.318434 5944 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:43.318610 5946 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:20:43.318706 5946 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:20:43.318696 5397 catalog_manager.cc:5495] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } health_report { overall_health: HEALTHY } } }
I20260925 07:20:43.325495 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.322547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.4.47.3:41237)
0925 07:20:43.322595 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.322597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.322705 (+ 108us) server_negotiation.cc:407] Connection header received
0925 07:20:43.322781 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.322782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.322798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.322862 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:43.323016 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.323564 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.324683 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.324881 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.324941 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.325194 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.325288 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.325344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.325422 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.325439 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:43.325562 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.322489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41237 (local address 127.0.0.1:33406)
0925 07:20:43.322622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.322635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.322700 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.322880 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.322881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.322883 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:43.322950 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.322953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.323652 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.323653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.324585 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.324588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.324958 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.324959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.325046 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.325065 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.325352 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.325361 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.325420 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.325476 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 07:20:43.326705 5695 raft_consensus.cc:1261] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Refusing update from remote peer 7fa49e70cf8042d1828acdb94aa39854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.326787 5556 raft_consensus.cc:1261] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Refusing update from remote peer 7fa49e70cf8042d1828acdb94aa39854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.326977 5944 consensus_queue.cc:1035] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.327152 5913 consensus_queue.cc:1035] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.327628 5910 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: TxnStatusTablet state changed. Reason: New leader 7fa49e70cf8042d1828acdb94aa39854. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.327721 5910 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.327965 5915 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: TxnStatusTablet state changed. Reason: New leader 7fa49e70cf8042d1828acdb94aa39854. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.328044 5915 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.328557 5913 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.328655 5913 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:43.328742 5913 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.328788 5913 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P 7fa49e70cf8042d1828acdb94aa39854: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:43.329017 5918 mvcc.cc:205] Tried to move back new op lower bound from 7333154174264184832 to 7333154174230810624. Current Snapshot: MvccSnapshot[applied={T|T < 7333154174264184832}]
I20260925 07:20:43.329107 5915 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.329174 5915 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P d5c16f59471341ed97744ed3513bfafa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.329255 5910 tablet_replica.cc:439] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fa49e70cf8042d1828acdb94aa39854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c16f59471341ed97744ed3513bfafa" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45175 } } peers { permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 } } peers { permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 } } }
I20260925 07:20:43.329332 5910 tablet_replica.cc:442] T 28af3f6e2ce4481d92bd48d145721b07 P a9f373c3e8094b5d8ffbc73c2620e88a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:43.334721 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.332260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:60563 (local address 127.4.47.3:41237)
0925 07:20:43.332301 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.332303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.332439 (+ 136us) server_negotiation.cc:407] Connection header received
0925 07:20:43.332506 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.332508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.332533 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.332599 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:43.332745 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.333237 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.334008 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.334215 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.334295 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.334465 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.334499 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.334563 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.334630 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.334649 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:20:43.334744 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.332169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41237 (local address 127.4.47.1:60563)
0925 07:20:43.332371 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.332381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.332455 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.332612 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.332613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.332614 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:43.332676 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.332678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.333320 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.333321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.333930 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.333931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.334297 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.334298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.334351 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.334366 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.334563 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.334565 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.334624 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.334671 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260925 07:20:43.336573 5836 raft_consensus.cc:1261] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Refusing update from remote peer d5c16f59471341ed97744ed3513bfafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.336720 5695 raft_consensus.cc:1261] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Refusing update from remote peer d5c16f59471341ed97744ed3513bfafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:43.336788 5911 consensus_queue.cc:1035] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa49e70cf8042d1828acdb94aa39854" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.336905 5911 consensus_queue.cc:1035] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f373c3e8094b5d8ffbc73c2620e88a" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:43.338181 5919 mvcc.cc:205] Tried to move back new op lower bound from 7333154174305296384 to 7333154173660590080. Current Snapshot: MvccSnapshot[applied={T|T < 7333154174305296384}]
I20260925 07:20:43.345009 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.342405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:59635 (local address 127.4.47.1:45175)
0925 07:20:43.342448 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.342450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.342458 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:43.342527 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.342528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.342539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.342595 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:43.342733 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.343283 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.344253 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.344544 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.344615 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.344778 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.344810 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.344857 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.344916 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.344935 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:43.345016 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.342346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45175 (local address 127.4.47.3:59635)
0925 07:20:43.342414 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:43.342427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:43.342467 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:43.342603 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:43.342603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:43.342604 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:43.342660 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.342661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.343370 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.343371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.344191 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:43.344193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.344615 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.344616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:43.344680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.344695 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:43.344857 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:43.344859 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:43.344910 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:20:43.344951 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 07:20:43.353868 5397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:43.353999 5397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:43.354641 5397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 415ead13e3704b38862780760ef2f72a: Sending DeleteTablet for 3 replicas of tablet 7e33f7b664cb4c0c94df1af3542b8b33
I20260925 07:20:43.354869 5536 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60030
I20260925 07:20:43.354945 5676 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260925 07:20:43.355085 5927 tablet_replica.cc:330] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: stopping tablet replica
I20260925 07:20:43.355099 5928 tablet_replica.cc:330] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: stopping tablet replica
I20260925 07:20:43.355149 5927 raft_consensus.cc:2229] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:43.355154 5928 raft_consensus.cc:2229] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:43.355192 5928 raft_consensus.cc:2258] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.355202 5927 raft_consensus.cc:2258] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.355530 5928 ts_tablet_manager.cc:1905] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.355634 5927 ts_tablet_manager.cc:1905] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.357110 5928 ts_tablet_manager.cc:1918] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:43.357218 5928 log.cc:1196] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/7e33f7b664cb4c0c94df1af3542b8b33
I20260925 07:20:43.357208 5816 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e33f7b664cb4c0c94df1af3542b8b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:43 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:33404
I20260925 07:20:43.357316 5929 tablet_replica.cc:330] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: stopping tablet replica
I20260925 07:20:43.357378 5929 raft_consensus.cc:2229] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:43.357419 5929 raft_consensus.cc:2258] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:43.357415 5927 ts_tablet_manager.cc:1918] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:43.357460 5928 ts_tablet_manager.cc:1939] T 7e33f7b664cb4c0c94df1af3542b8b33 P a9f373c3e8094b5d8ffbc73c2620e88a: Deleting consensus metadata
I20260925 07:20:43.357467 5927 log.cc:1196] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/7e33f7b664cb4c0c94df1af3542b8b33
I20260925 07:20:43.357671 5927 ts_tablet_manager.cc:1939] T 7e33f7b664cb4c0c94df1af3542b8b33 P d5c16f59471341ed97744ed3513bfafa: Deleting consensus metadata
I20260925 07:20:43.357774 5929 ts_tablet_manager.cc:1905] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:43.357786 5386 catalog_manager.cc:4841] TS a9f373c3e8094b5d8ffbc73c2620e88a (127.4.47.2:38301): tablet 7e33f7b664cb4c0c94df1af3542b8b33 (table test-table [id=3f921b8fbb8040a196ab85e06dd9b587]) successfully deleted
I20260925 07:20:43.357964 5384 catalog_manager.cc:4841] TS d5c16f59471341ed97744ed3513bfafa (127.4.47.1:45175): tablet 7e33f7b664cb4c0c94df1af3542b8b33 (table test-table [id=3f921b8fbb8040a196ab85e06dd9b587]) successfully deleted
I20260925 07:20:43.359301 5929 ts_tablet_manager.cc:1918] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:43.359352 5929 log.cc:1196] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/7e33f7b664cb4c0c94df1af3542b8b33
I20260925 07:20:43.359555 5929 ts_tablet_manager.cc:1939] T 7e33f7b664cb4c0c94df1af3542b8b33 P 7fa49e70cf8042d1828acdb94aa39854: Deleting consensus metadata
I20260925 07:20:43.359982 5386 catalog_manager.cc:4841] TS 7fa49e70cf8042d1828acdb94aa39854 (127.4.47.3:41237): tablet 7e33f7b664cb4c0c94df1af3542b8b33 (table test-table [id=3f921b8fbb8040a196ab85e06dd9b587]) successfully deleted
Sep 25 07:20:43 dist-test-slave-zw07 krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:43.366626 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.357511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.4.47.1:45175)
0925 07:20:43.357566 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.357568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.357919 (+ 351us) server_negotiation.cc:407] Connection header received
0925 07:20:43.358030 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.358031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.358044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.358105 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:43.358795 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.359377 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.360274 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.360473 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.360553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.363685 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:43.363694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:43.363694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:43.363706 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:43.365421 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.365809 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.365812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.365812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.365873 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.366114 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.366115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.366115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.366275 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:43.366387 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.366493 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.366561 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 07:20:43.366971 5596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60050
I20260925 07:20:43.374536 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:43.367977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.4.47.62:36005)
0925 07:20:43.368027 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:43.368029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:43.368038 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:20:43.368086 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:43.368086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:43.368095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:43.368139 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:43.368744 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.369327 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.370344 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:43.370536 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:43.370601 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:43.371931 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:43.371932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:43.371932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:43.371940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:43.373510 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.373830 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.373831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.373831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.373892 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:43.374061 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:43.374062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:43.374062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:43.374192 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:43.374300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:43.374364 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 07:20:43.374440 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:20:43.374753 5397 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37846
I20260925 07:20:43.375490 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5471
I20260925 07:20:43.384610 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5611
I20260925 07:20:43.390208 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5751
I20260925 07:20:43.395323 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5367
2026-09-25T07:20:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2287 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:43 dist-test-slave-zw07 krb5kdc[5967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:43 dist-test-slave-zw07 krb5kdc[5967](info): set up 2 sockets
Sep 25 07:20:43 dist-test-slave-zw07 krb5kdc[5967](info): commencing operation
krb5kdc: starting...
W20260925 07:20:44.435032 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:44 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:44.557665 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34169
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:44.638077 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:44.638285 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:44.638309 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:44.639407 5980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:44.639443 5980 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:20:44.639456 5980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:44.639468 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:44.639482 5980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:44.639492 5980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:44.640892 5980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:34169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:44.641098 5980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:44.641273 5980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:44.646147 5986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:44.646154 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:44.646250 5985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:44.646313 5980 server_base.cc:1034] running on GCE node
I20260925 07:20:44.646610 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:44.646900 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:44.648057 5980 hybrid_clock.cc:648] HybridClock initialized: now 1790320844648006 us; error 86 us; skew 500 ppm
Sep 25 07:20:44 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:44.650543 5980 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:44.650928 5980 webserver.cc:460] Webserver started at http://127.4.47.62:42345/ using document root <none> and password file <none>
I20260925 07:20:44.651175 5980 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:44.651226 5980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:44.651355 5980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:44.652410 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "665a58bd05414508b1935397561e7a6c"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "31ad46d25bf613d1c026fce5f6427837"
server_key_iv: "baf45b932fa00dc80fd4cfee5c75e2ac"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.652784 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "665a58bd05414508b1935397561e7a6c"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "31ad46d25bf613d1c026fce5f6427837"
server_key_iv: "baf45b932fa00dc80fd4cfee5c75e2ac"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.654134 5980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:44.654882 5995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:44.655049 5980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:44.655206 5980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "665a58bd05414508b1935397561e7a6c"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "31ad46d25bf613d1c026fce5f6427837"
server_key_iv: "baf45b932fa00dc80fd4cfee5c75e2ac"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.655262 5980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:44.665843 5980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:44.667877 5980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:44.667986 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:44.672397 5980 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:34169
I20260925 07:20:44.672488 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:34169 every 8 connection(s)
I20260925 07:20:44.672812 5980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:44.673497 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:44.675864 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Bootstrap starting.
I20260925 07:20:44.676402 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:44.676632 6058 log.cc:824] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:44.677313 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: No bootstrap required, opened a new log
I20260925 07:20:44.679425 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } }
I20260925 07:20:44.679567 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:44.679590 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665a58bd05414508b1935397561e7a6c, State: Initialized, Role: FOLLOWER
I20260925 07:20:44.679725 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } }
I20260925 07:20:44.679798 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:44.679827 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:44.679859 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:44.680346 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } }
I20260925 07:20:44.680431 6058 leader_election.cc:302] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665a58bd05414508b1935397561e7a6c; no voters:
I20260925 07:20:44.680560 6058 leader_election.cc:288] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:44.680639 6061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:44.680802 6061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [term 1 LEADER]: Becoming Leader. State: Replica: 665a58bd05414508b1935397561e7a6c, State: Running, Role: LEADER
I20260925 07:20:44.680833 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:44.680886 6061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } }
I20260925 07:20:44.681249 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665a58bd05414508b1935397561e7a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } } }
I20260925 07:20:44.681262 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665a58bd05414508b1935397561e7a6c. Latest consensus state: current_term: 1 leader_uuid: "665a58bd05414508b1935397561e7a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a58bd05414508b1935397561e7a6c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34169 } } }
I20260925 07:20:44.681350 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:44.681361 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:44.681555 6069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:44.682109 6069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:44.682848 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 5980
I20260925 07:20:44.682950 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:44.683189 4284 external_mini_cluster.cc:1428] Setting key 1b876cf871dc39fbea0cd6cfdc68521d
I20260925 07:20:44.683856 6069 catalog_manager.cc:1347] Generated new cluster ID: e22fe03b7e344f89a065e51a918c10d0
I20260925 07:20:44.683904 6069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:20:44 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:44.692524 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:44.683629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.4.47.62:34169)
0925 07:20:44.683775 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:44.683784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:44.683950 (+ 166us) server_negotiation.cc:407] Connection header received
0925 07:20:44.684109 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:44.684115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:44.684160 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:44.684289 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:44.684862 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.685769 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.686530 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.686977 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.687116 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:44.689343 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:44.689347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:44.689347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:44.689359 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:44.690971 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.691364 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.691367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.691368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.691463 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.691660 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.691661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.691662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.691905 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:44.692066 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:44.692173 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 07:20:44.692324 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260925 07:20:44.711186 6069 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:44.711712 6069 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:44.720376 6069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Generated new TSK 0
I20260925 07:20:44.720573 6069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:44.750958 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34169
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:44.830925 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:44.831151 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:44.831175 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:44.832307 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:44.832346 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:44.832366 6084 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:20:44.832409 6084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:44.832440 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:44.833840 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:44.834071 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:44.834288 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:44.839748 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:44.839748 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:44.839757 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:44.839967 6084 server_base.cc:1034] running on GCE node
I20260925 07:20:44.840160 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:44.840396 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:44.841558 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790320844841545 us; error 29 us; skew 500 ppm
Sep 25 07:20:44 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:44.844025 6084 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:44.844439 6084 webserver.cc:460] Webserver started at http://127.4.47.1:40015/ using document root <none> and password file <none>
I20260925 07:20:44.844657 6084 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:44.844720 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:44.844866 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:44.845942 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "c84ccfd04de04240b37d4662c372b614"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "b24cc8032b217f55e4a84b0514c40723"
server_key_iv: "36151ccab29954fa96349c2c506ffdb4"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.846297 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "c84ccfd04de04240b37d4662c372b614"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "b24cc8032b217f55e4a84b0514c40723"
server_key_iv: "36151ccab29954fa96349c2c506ffdb4"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.847675 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:44.848412 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:44.848569 6084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:44.848667 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "c84ccfd04de04240b37d4662c372b614"
format_stamp: "Formatted at 2026-09-25 07:20:44 on dist-test-slave-zw07"
server_key: "b24cc8032b217f55e4a84b0514c40723"
server_key_iv: "36151ccab29954fa96349c2c506ffdb4"
server_key_version: "encryptionkey@0"
I20260925 07:20:44.848737 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:44.865836 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:44.868093 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:44.868359 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:44.868886 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:44.868945 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:44.868986 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:44.869031 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:20:44 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:44.876255 6084 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:37407
I20260925 07:20:44.876328 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:37407 every 8 connection(s)
I20260925 07:20:44.876688 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:44.878906 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:44.869469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:36165 (local address 127.4.47.62:34169)
0925 07:20:44.869530 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:44.869532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:44.869770 (+ 238us) server_negotiation.cc:407] Connection header received
0925 07:20:44.870343 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:44.870345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:44.870361 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:44.870439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:44.871324 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.872216 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.873364 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.873585 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.873679 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:44.876104 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:44.876107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:44.876107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:44.876118 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:44.877734 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.878102 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.878102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.878103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.878152 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.878380 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.878380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.878381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.878486 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:44.878533 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:44.878771 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 07:20:44.878840 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 07:20:44.879127 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:44.869468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.1:36165)
0925 07:20:44.869653 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:44.869687 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:44.870198 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:44.870528 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:44.870533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:44.870953 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:44.871234 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:44.871239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.872357 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.872358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:44.873269 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:44.873272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.873706 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.873708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:44.873840 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:44.874355 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:44.874363 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:44.875988 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:44.877807 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:44.877812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:44.877814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:44.878035 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:44.878230 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:44.878230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:44.878230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:44.878327 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:44.878535 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:44.878540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:44.878660 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 07:20:44.878746 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:20:44.878868 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260925 07:20:44.879724 6215 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34169
I20260925 07:20:44.879819 6215 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:44.880024 6215 heartbeater.cc:511] Master 127.4.47.62:34169 requested a full tablet report, sending...
I20260925 07:20:44.880496 6012 ts_manager.cc:194] Registered new tserver with Master: c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407)
I20260925 07:20:44.881349 6012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:36165
I20260925 07:20:44.885825 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6084
I20260925 07:20:44.885967 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:44.886162 4284 external_mini_cluster.cc:1428] Setting key 9866e229010b557fce82612f3eee2d09
I20260925 07:20:44.886667 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:44.880895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46755 (local address 127.4.47.62:34169)
0925 07:20:44.880935 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:44.880937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:44.881011 (+ 74us) server_negotiation.cc:407] Connection header received
0925 07:20:44.881081 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:44.881081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:44.881091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:44.881153 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:44.881526 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.882043 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.882817 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.882996 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.883043 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:44.884295 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:44.884296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:44.884296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:44.884302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:44.885470 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.885832 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.885834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.885834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.885895 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:44.886110 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:44.886112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:44.886113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:44.886264 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:44.886340 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:44.886514 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 07:20:44.886604 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:20:44.886917 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:44.880840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.1:46755)
0925 07:20:44.880956 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:44.880967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:44.881023 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:44.881156 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:44.881156 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:44.881368 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:44.881465 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:44.881468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.882137 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.882138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:44.882744 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:44.882746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:44.883052 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:44.883053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:44.883154 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:44.883880 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:44.883885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:44.884169 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:44.885546 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:44.885547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:44.885547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:44.885726 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:44.885973 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:44.885974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:44.885974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:44.886051 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:44.886352 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:44.886353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:44.886405 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 07:20:44.886475 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:20:44.886595 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:44.912393 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34169
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:44.992826 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:44.993054 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:44.993098 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:44.994220 6224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:44.994282 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:44.994318 6224 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:20:44.994359 6224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:44.994397 6224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:44.995895 6224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:44.996165 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:44.996387 6224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:45.001336 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:45.001441 6224 server_base.cc:1034] running on GCE node
W20260925 07:20:45.001324 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:45.001302 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:45.001729 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:45.001987 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:45.003139 6224 hybrid_clock.cc:648] HybridClock initialized: now 1790320845003127 us; error 40 us; skew 500 ppm
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:45.005765 6224 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:45.006217 6224 webserver.cc:460] Webserver started at http://127.4.47.2:42067/ using document root <none> and password file <none>
I20260925 07:20:45.006482 6224 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:45.006531 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:45.006664 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:45.007835 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "d890cde091734ee099eae0f5754d2b1f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "f479be35ab88e331f329400708e96d19"
server_key_iv: "2dfb6c028be2f1463ec7fdc0e333c472"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.008201 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "d890cde091734ee099eae0f5754d2b1f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "f479be35ab88e331f329400708e96d19"
server_key_iv: "2dfb6c028be2f1463ec7fdc0e333c472"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.009593 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:45.010406 6239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.010612 6224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:45.010720 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "d890cde091734ee099eae0f5754d2b1f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "f479be35ab88e331f329400708e96d19"
server_key_iv: "2dfb6c028be2f1463ec7fdc0e333c472"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.010790 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:45.027304 6224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:45.029203 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:45.029357 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:45.029889 6224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:45.029948 6224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.029983 6224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:45.030005 6224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:45.037598 6224 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:40017
I20260925 07:20:45.037657 6354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:40017 every 8 connection(s)
I20260925 07:20:45.038007 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:45.040285 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.030593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56549 (local address 127.4.47.62:34169)
0925 07:20:45.030672 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.030674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.030927 (+ 253us) server_negotiation.cc:407] Connection header received
0925 07:20:45.031729 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.031730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.031742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.031827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.032508 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.033009 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.034034 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.034237 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.034321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.037230 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.037231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.037231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.037239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.038849 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.039372 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.039373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.039373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.039434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.039673 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.039674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.039674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.039788 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.039839 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.040117 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.040221 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:20:45.040356 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.030570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.2:56549)
0925 07:20:45.030773 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.030828 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.031615 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.031922 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.031926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.032225 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:45.032415 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.032420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.033100 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.033101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.033929 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.033931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.034330 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.034331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.034490 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.035009 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:45.035017 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:45.037114 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:45.038924 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.038929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.038931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.039269 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.039499 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.039500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.039500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.039597 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.039838 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:45.039842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:45.039960 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.040036 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.040166 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260925 07:20:45.041853 6355 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34169
I20260925 07:20:45.041940 6355 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:45.042135 6355 heartbeater.cc:511] Master 127.4.47.62:34169 requested a full tablet report, sending...
I20260925 07:20:45.042505 6011 ts_manager.cc:194] Registered new tserver with Master: d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017)
I20260925 07:20:45.042994 6011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:56549
I20260925 07:20:45.047429 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6224
I20260925 07:20:45.047535 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:45.047756 4284 external_mini_cluster.cc:1428] Setting key de53941f81a2c91bd9036a2d22c34733
I20260925 07:20:45.047819 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.042044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.2:56077)
0925 07:20:45.042240 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.042255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.042304 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.042520 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.042520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.042737 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:45.042792 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.042793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.043643 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.043643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.044177 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.044178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.044499 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.044500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.044556 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.045007 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:45.045012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:45.045332 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:45.046738 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.046738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.046739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.046897 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.047094 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.047095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.047095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.047151 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.047428 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:45.047430 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:45.047479 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.047595 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.047703 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260925 07:20:45.047816 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.042103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56077 (local address 127.4.47.62:34169)
0925 07:20:45.042164 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.042166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.042309 (+ 143us) server_negotiation.cc:407] Connection header received
0925 07:20:45.042444 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.042444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.042454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.042510 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.042861 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.043558 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.044256 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.044435 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.044491 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.045416 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.045417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.045417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.045424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.046636 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.046960 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.046961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.046961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.047011 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.047198 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.047199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.047199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.047361 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.047419 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.047649 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.047731 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:45.074111 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34169
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:45.157361 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:45.157598 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:45.157644 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:45.158857 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:45.158923 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:45.158959 6364 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:20:45.158989 6364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:20:45.159027 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:45.160496 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:45.160756 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:45.160982 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:45.165879 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:45.165910 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:45.165879 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:45.166078 6364 server_base.cc:1034] running on GCE node
I20260925 07:20:45.166289 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:45.166529 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:45.167680 6364 hybrid_clock.cc:648] HybridClock initialized: now 1790320845167673 us; error 33 us; skew 500 ppm
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:45.170073 6364 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:45.170454 6364 webserver.cc:460] Webserver started at http://127.4.47.3:34295/ using document root <none> and password file <none>
I20260925 07:20:45.170627 6364 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:45.170688 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:45.170782 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:45.171916 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "296bd2e3490c4030a915f683a4db3e5f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "80834d628a953bfd754cabf87b8d6dea"
server_key_iv: "f362bd7ba2f4d6b91f14ff046439ce6c"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.172209 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "296bd2e3490c4030a915f683a4db3e5f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "80834d628a953bfd754cabf87b8d6dea"
server_key_iv: "f362bd7ba2f4d6b91f14ff046439ce6c"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.173471 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:20:45.174120 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.174325 6364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:45.174410 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "296bd2e3490c4030a915f683a4db3e5f"
format_stamp: "Formatted at 2026-09-25 07:20:45 on dist-test-slave-zw07"
server_key: "80834d628a953bfd754cabf87b8d6dea"
server_key_iv: "f362bd7ba2f4d6b91f14ff046439ce6c"
server_key_version: "encryptionkey@0"
I20260925 07:20:45.174463 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:45.195164 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:45.197482 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:45.197643 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:45.198199 6364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:45.198251 6364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.198276 6364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:45.198321 6364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.205072 6364 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:42925
I20260925 07:20:45.205163 6494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:42925 every 8 connection(s)
I20260925 07:20:45.205461 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:45.208827 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.198767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:46217 (local address 127.4.47.62:34169)
0925 07:20:45.198831 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.198834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.199009 (+ 175us) server_negotiation.cc:407] Connection header received
0925 07:20:45.199562 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.199564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.199579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.199682 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.200580 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.201485 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.203103 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.203417 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.203502 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.206108 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.206109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.206109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.206116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.207554 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.207916 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.207917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.207917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.207967 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.208246 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.208247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.208247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.208371 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.208437 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.208686 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.208755 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:20:45.208990 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.198711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.3:46217)
0925 07:20:45.198872 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.198914 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.199468 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.199757 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.199760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.200144 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:45.200467 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.200475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.201595 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.201596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.202988 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.202991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.203511 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.203512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.203620 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.204306 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:45.204317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:45.206018 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:45.207629 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.207634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.207636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.207852 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.208021 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.208021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.208022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.208113 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.208438 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:45.208443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:45.208559 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.208668 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.208792 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260925 07:20:45.209031 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6364
I20260925 07:20:45.209132 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:45.209398 4284 external_mini_cluster.cc:1428] Setting key aaa96748a0bf11d75f6681d251a747c0
I20260925 07:20:45.209563 6495 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34169
I20260925 07:20:45.209646 6495 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:45.209856 6495 heartbeater.cc:511] Master 127.4.47.62:34169 requested a full tablet report, sending...
I20260925 07:20:45.210230 6011 ts_manager.cc:194] Registered new tserver with Master: 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:45.210628 6011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:46217
I20260925 07:20:45.215914 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.210613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:41683 (local address 127.4.47.62:34169)
0925 07:20:45.210652 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.210654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.210764 (+ 110us) server_negotiation.cc:407] Connection header received
0925 07:20:45.210823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.210823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.210833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.210900 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.211263 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.211706 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.212367 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.212540 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.212597 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.213387 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.213388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.213389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.213394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.214847 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.215170 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.215175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.215175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.215248 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.215453 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.215453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.215454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.215551 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.215603 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.215794 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.215854 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:20:45.215939 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.210554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.4.47.3:41683)
0925 07:20:45.210707 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.210717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.210776 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.210902 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.210903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.211136 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:45.211199 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.211201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.211811 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.211811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.212297 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.212298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.212610 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.212610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.212681 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.213041 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:45.213044 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:45.213312 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:45.214910 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.214911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.214911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.215060 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.215333 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.215334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.215335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.215397 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.215645 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:45.215646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:45.215684 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.215766 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.215859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":105,"threads_started":1}
I20260925 07:20:45.220999 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:45.243803 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.236105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.4.47.62:34169)
0925 07:20:45.236167 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.236169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.236197 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:20:45.236235 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.236236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.236245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.236301 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.236879 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.237392 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.238116 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.238343 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.238414 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.240946 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.240948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.240949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.240957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.242288 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.242663 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.242664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.242665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.242722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.242933 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.242934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.242935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.243049 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.243453 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.243612 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.243694 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:20:45.245322 6011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:45.246825 6011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:45.253228 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37407 (local address 127.0.0.1:48294)
0925 07:20:45.249267 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.249298 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.249367 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.249669 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.249673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.249676 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.249751 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.249752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.250925 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.250926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.252056 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.252058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.252942 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.252943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.253018 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.253032 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.253110 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.253157 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 07:20:45.253268 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.4.47.1:37407)
0925 07:20:45.249477 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.249482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.249492 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:45.249532 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.249534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.249552 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.249658 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.249840 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.250853 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.252140 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.252849 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.252931 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.252999 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.253131 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.253172 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260925 07:20:45.253710 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.4.47.2:40017)
0925 07:20:45.249674 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.249678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.249946 (+ 268us) server_negotiation.cc:407] Connection header received
0925 07:20:45.249990 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.249992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.250006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.250078 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.250220 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.251415 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.252641 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.253291 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.253360 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.253423 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.253602 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.253639 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 07:20:45.253947 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.0.0.1:44418)
0925 07:20:45.249888 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.249897 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.249938 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.250336 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.250336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.250338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.250369 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.250370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.251581 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.251582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.252582 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.252585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.253713 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.253714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.253771 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.253775 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.253822 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.253860 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260925 07:20:45.254012 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40017 (local address 127.0.0.1:58698)
0925 07:20:45.249814 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.249824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.249878 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.250088 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.250089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.250091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.250137 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.250139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.251505 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.251506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.252563 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.252565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.253453 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.253454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.253521 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.253524 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.253855 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.253899 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":29,"thread_start_us":62,"threads_started":1}
I20260925 07:20:45.253933 6149 tablet_service.cc:1467] Processing CreateTablet for tablet e29907b228984cf8b2c28632fa77f71d (DEFAULT_TABLE table=test-table [id=7bf0cc1609204123969d49cee5dd63bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.254328 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29907b228984cf8b2c28632fa77f71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.254338 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.249728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.4.47.3:42925)
0925 07:20:45.250166 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.250169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.250195 (+ 26us) server_negotiation.cc:407] Connection header received
0925 07:20:45.250233 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.250235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.250250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.250331 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.250431 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.251516 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.252689 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.253562 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.253623 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.253680 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.254197 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.254235 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":83,"threads_started":1}
I20260925 07:20:45.254652 6289 tablet_service.cc:1467] Processing CreateTablet for tablet e29907b228984cf8b2c28632fa77f71d (DEFAULT_TABLE table=test-table [id=7bf0cc1609204123969d49cee5dd63bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.254969 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29907b228984cf8b2c28632fa77f71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.257266 6518 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Bootstrap starting.
I20260925 07:20:45.257633 6429 tablet_service.cc:1467] Processing CreateTablet for tablet e29907b228984cf8b2c28632fa77f71d (DEFAULT_TABLE table=test-table [id=7bf0cc1609204123969d49cee5dd63bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.257958 6518 tablet_bootstrap.cc:654] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.257939 6429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e29907b228984cf8b2c28632fa77f71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.258277 6518 log.cc:824] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:45.258987 6518 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: No bootstrap required, opened a new log
I20260925 07:20:45.259078 6518 ts_tablet_manager.cc:1397] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:45.261132 6519 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Bootstrap starting.
I20260925 07:20:45.261168 6521 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Bootstrap starting.
I20260925 07:20:45.261283 6518 raft_consensus.cc:348] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.261463 6518 raft_consensus.cc:374] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.261507 6518 raft_consensus.cc:729] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.261633 6518 consensus_queue.cc:260] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.261744 6519 tablet_bootstrap.cc:654] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.261963 6215 heartbeater.cc:503] Master 127.4.47.62:34169 was elected leader, sending a full tablet report...
I20260925 07:20:45.262058 6519 log.cc:824] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:45.262136 6521 tablet_bootstrap.cc:654] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.262279 6518 ts_tablet_manager.cc:1428] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:20:45.262454 6521 log.cc:824] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:45.262878 6519 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: No bootstrap required, opened a new log
I20260925 07:20:45.262949 6519 ts_tablet_manager.cc:1397] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:45.263248 6521 tablet_bootstrap.cc:492] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: No bootstrap required, opened a new log
I20260925 07:20:45.263312 6521 ts_tablet_manager.cc:1397] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:45.264953 6519 raft_consensus.cc:348] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.265230 6519 raft_consensus.cc:374] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.265149 6521 raft_consensus.cc:348] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.265285 6519 raft_consensus.cc:729] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.265300 6521 raft_consensus.cc:374] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.265332 6521 raft_consensus.cc:729] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.265399 6519 consensus_queue.cc:260] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.265457 6521 consensus_queue.cc:260] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.265666 6519 ts_tablet_manager.cc:1428] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:20:45.265683 6521 ts_tablet_manager.cc:1428] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:45.265687 6355 heartbeater.cc:503] Master 127.4.47.62:34169 was elected leader, sending a full tablet report...
I20260925 07:20:45.265730 6495 heartbeater.cc:503] Master 127.4.47.62:34169 was elected leader, sending a full tablet report...
W20260925 07:20:45.289215 6356 tablet.cc:2369] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:20:45.377712 6216 tablet.cc:2369] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:20:45.456084 6496 tablet.cc:2369] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:45.608289 6522 raft_consensus.cc:482] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:45.608474 6522 raft_consensus.cc:504] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.608835 6522 leader_election.cc:288] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:45.613032 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.609119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:43771 (local address 127.4.47.2:40017)
0925 07:20:45.609236 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.609239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.609260 (+ 21us) server_negotiation.cc:407] Connection header received
0925 07:20:45.609367 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.609369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.609381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.609414 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.609639 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.610404 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612095 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.612638 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612725 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.612738 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.612923 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.612963 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 07:20:45.613032 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.609361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:58167 (local address 127.4.47.3:42925)
0925 07:20:45.609428 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.609430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.609438 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:45.609495 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.609496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.609507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.609561 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.609751 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.610431 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612091 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.612682 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612744 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.612753 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.612924 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.612962 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:20:45.613106 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.609228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.4.47.1:58167)
0925 07:20:45.609321 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.609334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.609405 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.609607 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.609608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.609611 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.609674 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.609676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.610502 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.610504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.611999 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.612002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.612751 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.612834 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.612841 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.612911 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.612968 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":60,"thread_start_us":45,"threads_started":1}
I20260925 07:20:45.613157 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.609020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40017 (local address 127.4.47.1:43771)
0925 07:20:45.609167 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.609179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.609273 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.609492 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.609492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.609495 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.609573 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.609575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.610505 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.610506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.611999 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.612002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.612744 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.612746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.612834 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.612843 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.612911 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.612968 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":45,"thread_start_us":97,"threads_started":1}
I20260925 07:20:45.613481 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29907b228984cf8b2c28632fa77f71d" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f" is_pre_election: true
I20260925 07:20:45.613478 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29907b228984cf8b2c28632fa77f71d" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f" is_pre_election: true
I20260925 07:20:45.613646 6309 raft_consensus.cc:2454] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 0.
I20260925 07:20:45.613659 6449 raft_consensus.cc:2454] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 0.
I20260925 07:20:45.613885 6100 leader_election.cc:302] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, c84ccfd04de04240b37d4662c372b614; no voters:
I20260925 07:20:45.614042 6522 raft_consensus.cc:2781] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:45.614115 6522 raft_consensus.cc:482] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:45.614135 6522 raft_consensus.cc:3037] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.614897 6522 raft_consensus.cc:504] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.615037 6522 leader_election.cc:288] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 election: Requested vote from peers d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:45.615263 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29907b228984cf8b2c28632fa77f71d" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f"
I20260925 07:20:45.615252 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e29907b228984cf8b2c28632fa77f71d" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f"
I20260925 07:20:45.615331 6309 raft_consensus.cc:3037] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.615331 6449 raft_consensus.cc:3037] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.616156 6449 raft_consensus.cc:2454] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 1.
I20260925 07:20:45.616156 6309 raft_consensus.cc:2454] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 1.
I20260925 07:20:45.616322 6100 leader_election.cc:302] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84ccfd04de04240b37d4662c372b614, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:45.616533 6522 raft_consensus.cc:2781] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:45.616671 6522 raft_consensus.cc:686] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 1 LEADER]: Becoming Leader. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Running, Role: LEADER
I20260925 07:20:45.616793 6522 consensus_queue.cc:237] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.617636 6011 catalog_manager.cc:5495] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 reported cstate change: term changed from 0 to 1, leader changed from <none> to c84ccfd04de04240b37d4662c372b614 (127.4.47.1). New cstate: current_term: 1 leader_uuid: "c84ccfd04de04240b37d4662c372b614" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
Sep 25 07:20:45 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:45.710635 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.703753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34169 (local address 127.0.0.1:43386)
0925 07:20:45.703939 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.703951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.704042 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.704207 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.704207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.704525 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:45.704569 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.704572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.705231 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.705232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.705768 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.705770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.706118 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.706118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.706169 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.706698 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:45.706703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:45.708077 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:45.709647 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.709666 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.709668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.709857 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.710025 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:45.710025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:45.710026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:45.710078 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:45.710322 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:45.710331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:45.710387 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.710444 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.710533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260925 07:20:45.710692 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.703832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.4.47.62:34169)
0925 07:20:45.703925 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.703927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.704024 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:20:45.704123 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.704124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.704134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.704201 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:45.704653 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.705142 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.705853 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.706038 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.706117 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.708160 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:45.708161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:45.708161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:45.708167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:45.709563 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.709918 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.709919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.709919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.709968 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:45.710116 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:45.710117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:45.710117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:45.710208 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:45.710275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.710474 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.710580 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":61}
I20260925 07:20:45.712558 6011 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:43386:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:20:45.712741 6011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:45.714267 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5d316c7d894f77bdb4af0abd55505b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c596f663bd0249ca9387496083fab0b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:45.714259 6289 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5d316c7d894f77bdb4af0abd55505b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c596f663bd0249ca9387496083fab0b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:45.714325 6429 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5d316c7d894f77bdb4af0abd55505b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c596f663bd0249ca9387496083fab0b8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:20:45.714430 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5d316c7d894f77bdb4af0abd55505b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.714440 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5d316c7d894f77bdb4af0abd55505b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.714447 6429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5d316c7d894f77bdb4af0abd55505b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.715696 6518 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: Bootstrap starting.
I20260925 07:20:45.715931 6519 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: Bootstrap starting.
I20260925 07:20:45.716133 6518 tablet_bootstrap.cc:654] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.716360 6519 tablet_bootstrap.cc:654] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.716676 6521 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: Bootstrap starting.
I20260925 07:20:45.716845 6518 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: No bootstrap required, opened a new log
I20260925 07:20:45.716883 6518 ts_tablet_manager.cc:1397] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:45.716995 6519 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: No bootstrap required, opened a new log
I20260925 07:20:45.717041 6521 tablet_bootstrap.cc:654] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.717038 6519 ts_tablet_manager.cc:1397] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:45.717170 6518 raft_consensus.cc:348] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.717224 6518 raft_consensus.cc:374] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.717237 6518 raft_consensus.cc:729] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.717206 6519 raft_consensus.cc:348] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.717273 6519 raft_consensus.cc:374] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.717299 6519 raft_consensus.cc:729] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.717293 6518 consensus_queue.cc:260] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.717347 6519 consensus_queue.cc:260] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.717404 6518 ts_tablet_manager.cc:1428] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:45.717463 6519 ts_tablet_manager.cc:1428] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.717504 6522 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.717519 6525 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.717581 6522 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.717607 6525 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.717830 6521 tablet_bootstrap.cc:492] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: No bootstrap required, opened a new log
I20260925 07:20:45.717880 6521 ts_tablet_manager.cc:1397] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:45.718022 6521 raft_consensus.cc:348] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.718091 6521 raft_consensus.cc:374] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.718111 6521 raft_consensus.cc:729] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.718158 6521 consensus_queue.cc:260] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.718245 6521 ts_tablet_manager.cc:1428] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.718295 6526 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.718398 6526 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.879848 6526 raft_consensus.cc:482] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:45.879938 6526 raft_consensus.cc:504] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.880306 6526 leader_election.cc:288] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017)
I20260925 07:20:45.883579 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.880844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:43563 (local address 127.4.47.2:40017)
0925 07:20:45.880887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.880889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.880902 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:20:45.880994 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.880995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.881005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.881061 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.881260 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.881827 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.882806 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.883208 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.883286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.883295 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.883481 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.883516 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:20:45.883603 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.880712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40017 (local address 127.4.47.3:43563)
0925 07:20:45.880850 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.880859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.880914 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.881084 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.881085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.881087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.881190 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.881191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.881903 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.881904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.882719 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.882721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.883329 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.883330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.883397 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.883400 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.883472 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.883512 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":46,"thread_start_us":52,"threads_started":1}
I20260925 07:20:45.883806 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.880507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37407 (local address 127.4.47.3:53625)
0925 07:20:45.880683 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.880694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.880799 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.881021 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.881021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.881024 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.881115 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.881117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.881827 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.881828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.882725 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.882727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.883537 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.883538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.883585 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.883588 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.883657 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.883704 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":46,"thread_start_us":79,"threads_started":1}
I20260925 07:20:45.883927 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.880566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:53625 (local address 127.4.47.1:37407)
0925 07:20:45.880707 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.880710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.880766 (+ 56us) server_negotiation.cc:407] Connection header received
0925 07:20:45.880920 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.880921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.880932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.881005 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.881189 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.881736 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.882815 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.883192 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.883269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.883283 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.883792 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.883838 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 07:20:45.883953 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5d316c7d894f77bdb4af0abd55505b" candidate_uuid: "296bd2e3490c4030a915f683a4db3e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f" is_pre_election: true
I20260925 07:20:45.884037 6309 raft_consensus.cc:2454] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296bd2e3490c4030a915f683a4db3e5f in term 0.
I20260925 07:20:45.884229 6380 leader_election.cc:302] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:45.884235 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5d316c7d894f77bdb4af0abd55505b" candidate_uuid: "296bd2e3490c4030a915f683a4db3e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614" is_pre_election: true
I20260925 07:20:45.884351 6169 raft_consensus.cc:2454] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296bd2e3490c4030a915f683a4db3e5f in term 0.
I20260925 07:20:45.884372 6526 raft_consensus.cc:2781] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:45.884411 6526 raft_consensus.cc:482] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:45.884445 6526 raft_consensus.cc:3037] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.885119 6526 raft_consensus.cc:504] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.885258 6526 leader_election.cc:288] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [CANDIDATE]: Term 1 election: Requested vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017)
I20260925 07:20:45.885394 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5d316c7d894f77bdb4af0abd55505b" candidate_uuid: "296bd2e3490c4030a915f683a4db3e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614"
I20260925 07:20:45.885464 6169 raft_consensus.cc:3037] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.885434 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5d316c7d894f77bdb4af0abd55505b" candidate_uuid: "296bd2e3490c4030a915f683a4db3e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f"
I20260925 07:20:45.885524 6309 raft_consensus.cc:3037] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.886090 6169 raft_consensus.cc:2454] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296bd2e3490c4030a915f683a4db3e5f in term 1.
I20260925 07:20:45.886150 6309 raft_consensus.cc:2454] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296bd2e3490c4030a915f683a4db3e5f in term 1.
I20260925 07:20:45.886273 6380 leader_election.cc:302] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, c84ccfd04de04240b37d4662c372b614; no voters:
I20260925 07:20:45.886382 6526 raft_consensus.cc:2781] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:45.886515 6526 raft_consensus.cc:686] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [term 1 LEADER]: Becoming Leader. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Running, Role: LEADER
I20260925 07:20:45.886608 6526 consensus_queue.cc:237] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.887032 6543 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: TxnStatusTablet state changed. Reason: New leader 296bd2e3490c4030a915f683a4db3e5f. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.887126 6543 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:45.887316 6545 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:20:45.887437 6545 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:20:45.887516 6010 catalog_manager.cc:5495] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3). New cstate: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } health_report { overall_health: HEALTHY } } }
I20260925 07:20:45.900501 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.897872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.4.47.3:42925)
0925 07:20:45.897921 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.897923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.897932 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:20:45.898014 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.898014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.898029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.898092 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:45.898233 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.898702 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.899820 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.899993 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.900044 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.900221 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.900309 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.900367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.900427 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.900445 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:20:45.900557 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.897803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.0.0.1:44428)
0925 07:20:45.897881 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.897892 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.897935 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.898113 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.898113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.898115 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:45.898168 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.898171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.898770 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.898771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.899744 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.899747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.900059 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.900060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.900120 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.900136 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.900368 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.900374 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.900426 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.900476 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 07:20:45.902420 6309 raft_consensus.cc:1261] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Refusing update from remote peer 296bd2e3490c4030a915f683a4db3e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:45.902423 6169 raft_consensus.cc:1261] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Refusing update from remote peer 296bd2e3490c4030a915f683a4db3e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:45.902777 6526 consensus_queue.cc:1035] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:45.902912 6543 consensus_queue.cc:1035] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:45.904134 6522 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: TxnStatusTablet state changed. Reason: New leader 296bd2e3490c4030a915f683a4db3e5f. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.904170 6525 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: TxnStatusTablet state changed. Reason: New leader 296bd2e3490c4030a915f683a4db3e5f. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.904255 6522 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.904264 6525 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.905234 6526 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.905316 6526 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:45.905421 6525 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.905509 6525 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P d890cde091734ee099eae0f5754d2b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.905808 6548 mvcc.cc:205] Tried to move back new op lower bound from 7333154184813649920 to 7333154184752144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333154184813649920 or (T in {7333154184813649920})}]
I20260925 07:20:45.906275 6550 mvcc.cc:205] Tried to move back new op lower bound from 7333154184813649920 to 7333154184752144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333154184813649920}]
I20260925 07:20:45.906467 6543 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.906551 6543 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P 296bd2e3490c4030a915f683a4db3e5f: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:20:45.906941 6522 tablet_replica.cc:439] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "296bd2e3490c4030a915f683a4db3e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } } }
I20260925 07:20:45.907029 6522 tablet_replica.cc:442] T 8e5d316c7d894f77bdb4af0abd55505b P c84ccfd04de04240b37d4662c372b614: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:20:45.910524 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.907584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.4.47.1:37407)
0925 07:20:45.907659 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.907661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.907673 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:20:45.907696 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.907697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.907711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.907765 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:45.907914 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.908648 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.909833 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.910015 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.910071 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.910256 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.910344 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.910403 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.910462 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.910478 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:20:45.915468 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.912952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:58245 (local address 127.4.47.3:42925)
0925 07:20:45.913060 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.913062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.913118 (+ 56us) server_negotiation.cc:407] Connection header received
0925 07:20:45.913187 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.913188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.913199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.913246 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:45.913373 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.913845 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.914752 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.914935 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.915013 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.915230 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.915263 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.915321 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.915380 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.915398 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 07:20:45.915500 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.912893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.4.47.1:58245)
0925 07:20:45.913066 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.913077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.913133 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.913248 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.913249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.913250 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:45.913308 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.913309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.913921 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.913922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.914674 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.914676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.915015 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.915016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.915101 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.915115 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.915321 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.915323 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.915375 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.915414 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260925 07:20:45.917783 6309 raft_consensus.cc:1261] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Refusing update from remote peer c84ccfd04de04240b37d4662c372b614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:45.917871 6449 raft_consensus.cc:1261] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Refusing update from remote peer c84ccfd04de04240b37d4662c372b614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:45.918035 6529 consensus_queue.cc:1035] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:45.918144 6522 consensus_queue.cc:1035] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:45.919773 6549 mvcc.cc:205] Tried to move back new op lower bound from 7333154184877654016 to 7333154183647076352. Current Snapshot: MvccSnapshot[applied={T|T < 7333154184877654016}]
I20260925 07:20:45.927307 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.924325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37407 (local address 127.4.47.3:38413)
0925 07:20:45.924471 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.924482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.924538 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.924649 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.924649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.924650 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:45.924696 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.924697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.925562 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.925563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.926472 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.926474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.926916 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.926917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.926958 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.926970 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.927131 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.927132 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.927208 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.927239 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 07:20:45.927309 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.924347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:38413 (local address 127.4.47.1:37407)
0925 07:20:45.924386 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.924388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.924544 (+ 156us) server_negotiation.cc:407] Connection header received
0925 07:20:45.924581 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.924582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.924591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.924647 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:45.924732 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.925475 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.926541 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.926848 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.926918 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.927032 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.927078 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.927122 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.927217 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.927237 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:20:45.936187 6012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:45.936296 6012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:45.937151 6012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Sending DeleteTablet for 3 replicas of tablet e29907b228984cf8b2c28632fa77f71d
I20260925 07:20:45.937469 6289 tablet_service.cc:1514] Processing DeleteTablet for tablet e29907b228984cf8b2c28632fa77f71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:45 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260925 07:20:45.937512 6429 tablet_service.cc:1514] Processing DeleteTablet for tablet e29907b228984cf8b2c28632fa77f71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:45 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44418
I20260925 07:20:45.937669 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet e29907b228984cf8b2c28632fa77f71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:45 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260925 07:20:45.937883 6571 tablet_replica.cc:330] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: stopping tablet replica
I20260925 07:20:45.937985 6571 raft_consensus.cc:2229] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:45.938072 6571 raft_consensus.cc:2258] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:45.938318 6012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:45.938453 6012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:45.938464 6571 ts_tablet_manager.cc:1905] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:45.939920 6429 tablet_service.cc:1467] Processing CreateTablet for tablet f76ef16951ff4954b5941747d9b65325 (DEFAULT_TABLE table=test-table [id=4f4d9310eee64d7996a6e6672f80b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.940045 6429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76ef16951ff4954b5941747d9b65325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.940320 6571 ts_tablet_manager.cc:1918] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:45.940384 6571 log.cc:1196] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/e29907b228984cf8b2c28632fa77f71d
I20260925 07:20:45.940505 6289 tablet_service.cc:1467] Processing CreateTablet for tablet f76ef16951ff4954b5941747d9b65325 (DEFAULT_TABLE table=test-table [id=4f4d9310eee64d7996a6e6672f80b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.940647 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76ef16951ff4954b5941747d9b65325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.940672 6571 ts_tablet_manager.cc:1939] T e29907b228984cf8b2c28632fa77f71d P c84ccfd04de04240b37d4662c372b614: Deleting consensus metadata
I20260925 07:20:45.941054 5996 catalog_manager.cc:4841] TS c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407): tablet e29907b228984cf8b2c28632fa77f71d (table test-table [id=7bf0cc1609204123969d49cee5dd63bf]) successfully deleted
I20260925 07:20:45.942011 6519 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Bootstrap starting.
I20260925 07:20:45.942147 6149 tablet_service.cc:1467] Processing CreateTablet for tablet f76ef16951ff4954b5941747d9b65325 (DEFAULT_TABLE table=test-table [id=4f4d9310eee64d7996a6e6672f80b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:45.942242 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76ef16951ff4954b5941747d9b65325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:45.942413 6519 tablet_bootstrap.cc:654] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.943446 6518 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Bootstrap starting.
I20260925 07:20:45.943459 6519 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: No bootstrap required, opened a new log
I20260925 07:20:45.943500 6519 ts_tablet_manager.cc:1397] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:45.943635 6519 raft_consensus.cc:348] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.943714 6519 raft_consensus.cc:374] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.943737 6519 raft_consensus.cc:729] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.943794 6519 consensus_queue.cc:260] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.943845 6518 tablet_bootstrap.cc:654] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.943878 6519 ts_tablet_manager.cc:1428] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:45.945596 6521 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Bootstrap starting.
I20260925 07:20:45.945993 6521 tablet_bootstrap.cc:654] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:45.946641 6521 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: No bootstrap required, opened a new log
I20260925 07:20:45.946689 6521 ts_tablet_manager.cc:1397] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:45.946831 6521 raft_consensus.cc:348] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.946905 6521 raft_consensus.cc:374] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.946933 6521 raft_consensus.cc:729] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.946969 6521 consensus_queue.cc:260] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.947101 6521 ts_tablet_manager.cc:1428] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:45.947319 6518 tablet_bootstrap.cc:492] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: No bootstrap required, opened a new log
I20260925 07:20:45.947365 6518 ts_tablet_manager.cc:1397] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 07:20:45.947492 6518 raft_consensus.cc:348] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.947551 6518 raft_consensus.cc:374] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:45.947571 6518 raft_consensus.cc:729] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Initialized, Role: FOLLOWER
I20260925 07:20:45.947624 6518 consensus_queue.cc:260] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.947702 6518 ts_tablet_manager.cc:1428] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:45.948565 6569 tablet_replica.cc:330] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: stopping tablet replica
I20260925 07:20:45.948771 6569 raft_consensus.cc:2229] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:45.948882 6569 raft_consensus.cc:2258] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:45.949313 6569 ts_tablet_manager.cc:1905] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:45.951051 6570 tablet_replica.cc:330] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: stopping tablet replica
I20260925 07:20:45.951171 6569 ts_tablet_manager.cc:1918] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:45.951251 6569 log.cc:1196] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/e29907b228984cf8b2c28632fa77f71d
I20260925 07:20:45.951350 6570 raft_consensus.cc:2229] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:45.951459 6570 raft_consensus.cc:2258] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:45.951507 6569 ts_tablet_manager.cc:1939] T e29907b228984cf8b2c28632fa77f71d P d890cde091734ee099eae0f5754d2b1f: Deleting consensus metadata
I20260925 07:20:45.951967 6570 ts_tablet_manager.cc:1905] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:45.951984 5996 catalog_manager.cc:4841] TS d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017): tablet e29907b228984cf8b2c28632fa77f71d (table test-table [id=7bf0cc1609204123969d49cee5dd63bf]) successfully deleted
I20260925 07:20:45.954107 6570 ts_tablet_manager.cc:1918] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:45.954232 6570 log.cc:1196] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/e29907b228984cf8b2c28632fa77f71d
I20260925 07:20:45.954527 6570 ts_tablet_manager.cc:1939] T e29907b228984cf8b2c28632fa77f71d P 296bd2e3490c4030a915f683a4db3e5f: Deleting consensus metadata
I20260925 07:20:45.954986 5996 catalog_manager.cc:4841] TS 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925): tablet e29907b228984cf8b2c28632fa77f71d (table test-table [id=7bf0cc1609204123969d49cee5dd63bf]) successfully deleted
I20260925 07:20:45.981223 6525 raft_consensus.cc:482] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:45.981348 6525 raft_consensus.cc:504] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.981696 6525 leader_election.cc:288] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:45.985968 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.982403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:46107 (local address 127.4.47.1:37407)
0925 07:20:45.982455 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.982457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.982465 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:45.982492 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.982493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.982501 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.982548 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.982765 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.983519 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.985002 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.985590 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.985636 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.985648 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.985860 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.985894 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:20:45.986234 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.981946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37407 (local address 127.4.47.2:46107)
0925 07:20:45.982084 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.982100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.982170 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.982610 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.982611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.982613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.982687 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.982689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.983629 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.983630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.984892 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.984895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.985707 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.985708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.985781 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.985785 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.986108 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.986152 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260925 07:20:45.986555 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76ef16951ff4954b5941747d9b65325" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614" is_pre_election: true
I20260925 07:20:45.986663 6169 raft_consensus.cc:2454] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 0.
I20260925 07:20:45.986842 6240 leader_election.cc:302] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84ccfd04de04240b37d4662c372b614, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:45.986892 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.982518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:53675 (local address 127.4.47.3:42925)
0925 07:20:45.982564 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.982566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.982575 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:20:45.982602 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.982603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.982614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.982684 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:45.983327 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.984029 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.985374 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.985917 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.985967 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.985977 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.986784 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.986823 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:45.986961 6525 raft_consensus.cc:2781] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:45.986991 6525 raft_consensus.cc:482] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:45.987012 6525 raft_consensus.cc:3037] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.987412 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.982159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.4.47.2:53675)
0925 07:20:45.982282 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:45.982290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:45.982324 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:45.982734 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:45.982735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:45.982737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:45.983225 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.983229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.984134 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.984135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.985264 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:45.985267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.986242 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.986242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:45.986574 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.986579 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:20:45.987286 (+ 707us) client_negotiation.cc:240] Negotiation successful
0925 07:20:45.987331 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 07:20:45.987763 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76ef16951ff4954b5941747d9b65325" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f" is_pre_election: true
I20260925 07:20:45.987848 6449 raft_consensus.cc:2454] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 0.
I20260925 07:20:45.987990 6525 raft_consensus.cc:504] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.988126 6525 leader_election.cc:288] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 election: Requested vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:45.988298 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76ef16951ff4954b5941747d9b65325" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f"
I20260925 07:20:45.988372 6449 raft_consensus.cc:3037] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.989136 6449 raft_consensus.cc:2454] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 1.
I20260925 07:20:45.989306 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76ef16951ff4954b5941747d9b65325" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614"
I20260925 07:20:45.989387 6169 raft_consensus.cc:3037] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:45.990113 6169 raft_consensus.cc:2454] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 1.
I20260925 07:20:45.990242 6240 leader_election.cc:302] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:45.990347 6525 raft_consensus.cc:2781] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:45.990492 6525 raft_consensus.cc:686] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 1 LEADER]: Becoming Leader. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Running, Role: LEADER
I20260925 07:20:45.990566 6525 consensus_queue.cc:237] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:45.991276 6012 catalog_manager.cc:5495] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d890cde091734ee099eae0f5754d2b1f (127.4.47.2). New cstate: current_term: 1 leader_uuid: "d890cde091734ee099eae0f5754d2b1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
I20260925 07:20:45.999682 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:45.996424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.4.47.2:40017)
0925 07:20:45.996486 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:45.996488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:45.996496 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:45.996532 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:45.996532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:45.996553 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:45.996639 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:45.996789 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.997527 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.998722 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:45.999022 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:45.999121 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:45.999306 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:45.999443 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:45.999515 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:45.999583 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:20:45.999603 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:20:46.006043 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:46.002621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:38701 (local address 127.4.47.3:42925)
0925 07:20:46.002662 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:46.002670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:46.002792 (+ 122us) server_negotiation.cc:407] Connection header received
0925 07:20:46.002846 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:46.002847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:46.002860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:46.002921 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:46.003093 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.003773 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.005057 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.005438 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.005514 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:46.005746 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:46.005791 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:46.005845 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:46.005928 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 07:20:46.005952 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:20:46.006105 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:46.002554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:42925 (local address 127.4.47.2:38701)
0925 07:20:46.002723 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:46.002734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:46.002793 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:46.002922 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:46.002922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:46.002924 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:46.003007 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:46.003009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.003882 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.003883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:46.004945 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:46.004948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.005522 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.005523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:46.005606 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:46.005627 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:46.005857 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:46.005866 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:20:46.005924 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:20:46.005974 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 07:20:46.008766 6449 raft_consensus.cc:1261] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Refusing update from remote peer d890cde091734ee099eae0f5754d2b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:46.009099 6525 consensus_queue.cc:1035] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:46.009330 6169 raft_consensus.cc:1261] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Refusing update from remote peer d890cde091734ee099eae0f5754d2b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:46.010550 6577 consensus_queue.cc:1035] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:46.019979 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:46.016291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40017 (local address 127.4.47.3:34999)
0925 07:20:46.016340 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:46.016354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:46.016403 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:46.016581 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:46.016581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:46.016583 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:20:46.016644 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:46.016645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.017963 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.017964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:46.018909 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:46.018912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.019512 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.019513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:46.019571 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:46.019579 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:20:46.019794 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:20:46.019796 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:46.019841 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:20:46.019889 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 07:20:46.020503 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:46.016395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:34999 (local address 127.4.47.2:40017)
0925 07:20:46.016434 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:46.016436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:46.016446 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:46.016471 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:46.016472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:46.016483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:46.016533 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:46.016817 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.017641 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.019020 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:46.019397 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:46.019472 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:46.019681 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:46.019718 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:46.019768 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:46.020402 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 07:20:46.020428 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:20:46.021015 6550 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 07:20:47.546452 6012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:47.546550 6012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:47.547276 6012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Sending DeleteTablet for 3 replicas of tablet f76ef16951ff4954b5941747d9b65325
I20260925 07:20:47.547530 6289 tablet_service.cc:1514] Processing DeleteTablet for tablet f76ef16951ff4954b5941747d9b65325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260925 07:20:47.547557 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet f76ef16951ff4954b5941747d9b65325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260925 07:20:47.547605 6428 tablet_service.cc:1514] Processing DeleteTablet for tablet f76ef16951ff4954b5941747d9b65325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44418
I20260925 07:20:47.547771 6609 tablet_replica.cc:330] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: stopping tablet replica
I20260925 07:20:47.547796 6608 tablet_replica.cc:330] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: stopping tablet replica
I20260925 07:20:47.547838 6609 raft_consensus.cc:2229] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:47.547842 6608 raft_consensus.cc:2229] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:47.547879 6609 raft_consensus.cc:2258] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.547879 6608 raft_consensus.cc:2258] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.548103 6609 ts_tablet_manager.cc:1905] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.548204 6608 ts_tablet_manager.cc:1905] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.550012 6608 ts_tablet_manager.cc:1918] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:20:47.550012 6609 ts_tablet_manager.cc:1918] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:20:47.550081 6608 log.cc:1196] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/f76ef16951ff4954b5941747d9b65325
I20260925 07:20:47.550081 6609 log.cc:1196] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/f76ef16951ff4954b5941747d9b65325
I20260925 07:20:47.550313 6608 ts_tablet_manager.cc:1939] T f76ef16951ff4954b5941747d9b65325 P c84ccfd04de04240b37d4662c372b614: Deleting consensus metadata
I20260925 07:20:47.550313 6609 ts_tablet_manager.cc:1939] T f76ef16951ff4954b5941747d9b65325 P 296bd2e3490c4030a915f683a4db3e5f: Deleting consensus metadata
I20260925 07:20:47.550338 6607 tablet_replica.cc:330] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: stopping tablet replica
I20260925 07:20:47.550420 6607 raft_consensus.cc:2229] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:47.550513 6607 raft_consensus.cc:2258] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.550618 5996 catalog_manager.cc:4841] TS 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925): tablet f76ef16951ff4954b5941747d9b65325 (table test-table [id=4f4d9310eee64d7996a6e6672f80b342]) successfully deleted
I20260925 07:20:47.550720 5996 catalog_manager.cc:4841] TS c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407): tablet f76ef16951ff4954b5941747d9b65325 (table test-table [id=4f4d9310eee64d7996a6e6672f80b342]) successfully deleted
I20260925 07:20:47.550802 6607 ts_tablet_manager.cc:1905] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.552294 6607 ts_tablet_manager.cc:1918] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:20:47.552361 6607 log.cc:1196] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/f76ef16951ff4954b5941747d9b65325
I20260925 07:20:47.552588 6607 ts_tablet_manager.cc:1939] T f76ef16951ff4954b5941747d9b65325 P d890cde091734ee099eae0f5754d2b1f: Deleting consensus metadata
I20260925 07:20:47.553030 5996 catalog_manager.cc:4841] TS d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017): tablet f76ef16951ff4954b5941747d9b65325 (table test-table [id=4f4d9310eee64d7996a6e6672f80b342]) successfully deleted
Sep 25 07:20:47 dist-test-slave-zw07 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:20:47 dist-test-slave-zw07 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:47.571041 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:47.564309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.4.47.62:34169)
0925 07:20:47.564455 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:47.564457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:47.564465 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:20:47.564524 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:47.564525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:47.564534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:47.564620 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:47.565196 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.565696 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.566423 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.566616 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.566665 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:47.568914 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:47.568915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:47.568915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:47.568921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:47.570142 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:47.570431 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:47.570432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:47.570432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:47.570481 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:47.570632 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:47.570633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:47.570633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:47.570740 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:47.570840 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:47.570911 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 07:20:47.570973 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260925 07:20:47.572484 6012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:47.572620 6012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:47.573864 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (DEFAULT_TABLE table=test-table [id=70f15a5bd28a486caba9b10941045e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.573930 6428 tablet_service.cc:1467] Processing CreateTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (DEFAULT_TABLE table=test-table [id=70f15a5bd28a486caba9b10941045e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.574004 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cba00e4dabc4f28b50028e7fe66d7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.574075 6428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cba00e4dabc4f28b50028e7fe66d7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.575153 6618 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Bootstrap starting.
I20260925 07:20:47.575178 6289 tablet_service.cc:1467] Processing CreateTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (DEFAULT_TABLE table=test-table [id=70f15a5bd28a486caba9b10941045e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.575284 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cba00e4dabc4f28b50028e7fe66d7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.575435 6619 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Bootstrap starting.
I20260925 07:20:47.575523 6618 tablet_bootstrap.cc:654] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.575794 6619 tablet_bootstrap.cc:654] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.576162 6618 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: No bootstrap required, opened a new log
I20260925 07:20:47.576272 6618 ts_tablet_manager.cc:1397] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:47.576316 6619 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: No bootstrap required, opened a new log
I20260925 07:20:47.576375 6619 ts_tablet_manager.cc:1397] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:47.576435 6618 raft_consensus.cc:348] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.576519 6618 raft_consensus.cc:374] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.576546 6618 raft_consensus.cc:729] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.576514 6619 raft_consensus.cc:348] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.576596 6619 raft_consensus.cc:374] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.576619 6619 raft_consensus.cc:729] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.576670 6619 consensus_queue.cc:260] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.576624 6618 consensus_queue.cc:260] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.576776 6619 ts_tablet_manager.cc:1428] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:47.576985 6618 ts_tablet_manager.cc:1428] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 07:20:47.577104 6622 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Bootstrap starting.
I20260925 07:20:47.577481 6622 tablet_bootstrap.cc:654] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.578085 6622 tablet_bootstrap.cc:492] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: No bootstrap required, opened a new log
I20260925 07:20:47.578126 6622 ts_tablet_manager.cc:1397] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:47.578239 6622 raft_consensus.cc:348] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.578297 6622 raft_consensus.cc:374] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.578310 6622 raft_consensus.cc:729] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.578361 6622 consensus_queue.cc:260] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.578454 6622 ts_tablet_manager.cc:1428] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:47.612099 6623 raft_consensus.cc:482] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:47.612190 6623 raft_consensus.cc:504] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.612323 6623 leader_election.cc:288] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:47.612510 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cba00e4dabc4f28b50028e7fe66d7e5" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f" is_pre_election: true
I20260925 07:20:47.612515 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cba00e4dabc4f28b50028e7fe66d7e5" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f" is_pre_election: true
I20260925 07:20:47.612622 6309 raft_consensus.cc:2454] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 0.
I20260925 07:20:47.612607 6449 raft_consensus.cc:2454] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 0.
I20260925 07:20:47.612782 6100 leader_election.cc:302] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84ccfd04de04240b37d4662c372b614, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:47.612895 6623 raft_consensus.cc:2781] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:47.612936 6623 raft_consensus.cc:482] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:47.612960 6623 raft_consensus.cc:3037] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.613653 6623 raft_consensus.cc:504] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.613770 6623 leader_election.cc:288] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 election: Requested vote from peers d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:47.613946 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cba00e4dabc4f28b50028e7fe66d7e5" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f"
I20260925 07:20:47.613947 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cba00e4dabc4f28b50028e7fe66d7e5" candidate_uuid: "c84ccfd04de04240b37d4662c372b614" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d890cde091734ee099eae0f5754d2b1f"
I20260925 07:20:47.614017 6449 raft_consensus.cc:3037] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.614017 6309 raft_consensus.cc:3037] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.614758 6449 raft_consensus.cc:2454] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 1.
I20260925 07:20:47.614758 6309 raft_consensus.cc:2454] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84ccfd04de04240b37d4662c372b614 in term 1.
I20260925 07:20:47.614943 6100 leader_election.cc:302] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, c84ccfd04de04240b37d4662c372b614; no voters:
I20260925 07:20:47.615039 6623 raft_consensus.cc:2781] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:47.615207 6623 raft_consensus.cc:686] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 1 LEADER]: Becoming Leader. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Running, Role: LEADER
I20260925 07:20:47.615250 6623 consensus_queue.cc:237] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.615844 6010 catalog_manager.cc:5495] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 reported cstate change: term changed from 0 to 1, leader changed from <none> to c84ccfd04de04240b37d4662c372b614 (127.4.47.1). New cstate: current_term: 1 leader_uuid: "c84ccfd04de04240b37d4662c372b614" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
I20260925 07:20:47.624797 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:47.622399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.4.47.1:37407)
0925 07:20:47.622554 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:47.622556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:47.622562 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:20:47.622582 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:47.622583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:47.622593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:47.622668 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:47.622810 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.623363 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.624101 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.624308 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.624388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:47.624538 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:47.624577 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:47.624634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:47.624687 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 07:20:47.624709 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260925 07:20:47.626878 6449 raft_consensus.cc:1261] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Refusing update from remote peer c84ccfd04de04240b37d4662c372b614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:47.626892 6309 raft_consensus.cc:1261] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Refusing update from remote peer c84ccfd04de04240b37d4662c372b614: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:47.627195 6623 consensus_queue.cc:1035] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:47.627287 6625 consensus_queue.cc:1035] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:47.637072 6010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:47.637164 6010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:47.637735 6010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Sending DeleteTablet for 3 replicas of tablet 2cba00e4dabc4f28b50028e7fe66d7e5
I20260925 07:20:47.637933 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260925 07:20:47.637961 6289 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260925 07:20:47.637977 6428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cba00e4dabc4f28b50028e7fe66d7e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:47 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44418
I20260925 07:20:47.638029 6608 tablet_replica.cc:330] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: stopping tablet replica
I20260925 07:20:47.638063 6607 tablet_replica.cc:330] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: stopping tablet replica
I20260925 07:20:47.638067 6609 tablet_replica.cc:330] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: stopping tablet replica
I20260925 07:20:47.638089 6608 raft_consensus.cc:2229] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:47.638127 6607 raft_consensus.cc:2229] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:47.638127 6609 raft_consensus.cc:2229] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:47.638141 6608 raft_consensus.cc:2258] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.638168 6607 raft_consensus.cc:2258] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.638167 6609 raft_consensus.cc:2258] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:47.638456 6608 ts_tablet_manager.cc:1905] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.638537 6609 ts_tablet_manager.cc:1905] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.638727 6607 ts_tablet_manager.cc:1905] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:47.638706 6010 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:47.638821 6010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:47.640129 6608 ts_tablet_manager.cc:1918] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:47.640204 6608 log.cc:1196] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/2cba00e4dabc4f28b50028e7fe66d7e5
I20260925 07:20:47.640300 6609 ts_tablet_manager.cc:1918] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:47.640353 6609 log.cc:1196] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/2cba00e4dabc4f28b50028e7fe66d7e5
I20260925 07:20:47.640354 6607 ts_tablet_manager.cc:1918] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:20:47.640396 6607 log.cc:1196] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/2cba00e4dabc4f28b50028e7fe66d7e5
I20260925 07:20:47.640471 6608 ts_tablet_manager.cc:1939] T 2cba00e4dabc4f28b50028e7fe66d7e5 P c84ccfd04de04240b37d4662c372b614: Deleting consensus metadata
I20260925 07:20:47.640570 6609 ts_tablet_manager.cc:1939] T 2cba00e4dabc4f28b50028e7fe66d7e5 P 296bd2e3490c4030a915f683a4db3e5f: Deleting consensus metadata
I20260925 07:20:47.640595 6607 ts_tablet_manager.cc:1939] T 2cba00e4dabc4f28b50028e7fe66d7e5 P d890cde091734ee099eae0f5754d2b1f: Deleting consensus metadata
I20260925 07:20:47.640959 5996 catalog_manager.cc:4841] TS c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407): tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (table test-table [id=70f15a5bd28a486caba9b10941045e37]) successfully deleted
I20260925 07:20:47.640983 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 (DEFAULT_TABLE table=test-table [id=1d5237951a9d4bb08f717719354801ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.641098 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2634a7bdf1c04f5abbc307158f7ad1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.641134 5996 catalog_manager.cc:4841] TS 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925): tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (table test-table [id=70f15a5bd28a486caba9b10941045e37]) successfully deleted
I20260925 07:20:47.641170 6428 tablet_service.cc:1467] Processing CreateTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 (DEFAULT_TABLE table=test-table [id=1d5237951a9d4bb08f717719354801ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.641263 6428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2634a7bdf1c04f5abbc307158f7ad1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.642103 6618 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614: Bootstrap starting.
I20260925 07:20:47.642148 6619 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Bootstrap starting.
I20260925 07:20:47.642247 6289 tablet_service.cc:1467] Processing CreateTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 (DEFAULT_TABLE table=test-table [id=1d5237951a9d4bb08f717719354801ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:47.642338 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2634a7bdf1c04f5abbc307158f7ad1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:47.642419 6619 tablet_bootstrap.cc:654] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.642446 6618 tablet_bootstrap.cc:654] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.642946 6619 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: No bootstrap required, opened a new log
I20260925 07:20:47.643003 6619 ts_tablet_manager.cc:1397] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 07:20:47.643132 6619 raft_consensus.cc:348] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.643203 6619 raft_consensus.cc:374] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.643221 6619 raft_consensus.cc:729] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296bd2e3490c4030a915f683a4db3e5f, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.643260 6619 consensus_queue.cc:260] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.643349 6619 ts_tablet_manager.cc:1428] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:47.643550 6618 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614: No bootstrap required, opened a new log
I20260925 07:20:47.643602 6618 ts_tablet_manager.cc:1397] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:47.643625 6622 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Bootstrap starting.
I20260925 07:20:47.643707 5996 catalog_manager.cc:4841] TS d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017): tablet 2cba00e4dabc4f28b50028e7fe66d7e5 (table test-table [id=70f15a5bd28a486caba9b10941045e37]) successfully deleted
I20260925 07:20:47.643702 6618 raft_consensus.cc:348] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.643782 6618 raft_consensus.cc:374] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.643807 6618 raft_consensus.cc:729] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ccfd04de04240b37d4662c372b614, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.643859 6618 consensus_queue.cc:260] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.643916 6622 tablet_bootstrap.cc:654] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:47.643950 6618 ts_tablet_manager.cc:1428] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:47.644464 6622 tablet_bootstrap.cc:492] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: No bootstrap required, opened a new log
I20260925 07:20:47.644515 6622 ts_tablet_manager.cc:1397] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:47.644634 6622 raft_consensus.cc:348] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.644704 6622 raft_consensus.cc:374] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:47.644724 6622 raft_consensus.cc:729] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Initialized, Role: FOLLOWER
I20260925 07:20:47.644762 6622 consensus_queue.cc:260] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.644848 6622 ts_tablet_manager.cc:1428] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:47.678061 6585 raft_consensus.cc:482] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:47.678165 6585 raft_consensus.cc:504] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.678340 6585 leader_election.cc:288] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:47.678689 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2634a7bdf1c04f5abbc307158f7ad1f8" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614" is_pre_election: true
I20260925 07:20:47.678781 6169 raft_consensus.cc:2454] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 0.
I20260925 07:20:47.678732 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2634a7bdf1c04f5abbc307158f7ad1f8" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f" is_pre_election: true
I20260925 07:20:47.678869 6449 raft_consensus.cc:2454] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 0.
I20260925 07:20:47.678953 6240 leader_election.cc:302] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c84ccfd04de04240b37d4662c372b614, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:47.679091 6585 raft_consensus.cc:2781] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:47.679141 6585 raft_consensus.cc:482] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:47.679178 6585 raft_consensus.cc:3037] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.679986 6585 raft_consensus.cc:504] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.680142 6585 leader_election.cc:288] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 election: Requested vote from peers c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407), 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925)
I20260925 07:20:47.680295 6449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2634a7bdf1c04f5abbc307158f7ad1f8" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296bd2e3490c4030a915f683a4db3e5f"
I20260925 07:20:47.680363 6449 raft_consensus.cc:3037] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.680306 6169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2634a7bdf1c04f5abbc307158f7ad1f8" candidate_uuid: "d890cde091734ee099eae0f5754d2b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ccfd04de04240b37d4662c372b614"
I20260925 07:20:47.680408 6169 raft_consensus.cc:3037] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:47.681033 6449 raft_consensus.cc:2454] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 1.
I20260925 07:20:47.681182 6169 raft_consensus.cc:2454] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d890cde091734ee099eae0f5754d2b1f in term 1.
I20260925 07:20:47.681208 6240 leader_election.cc:302] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 296bd2e3490c4030a915f683a4db3e5f, d890cde091734ee099eae0f5754d2b1f; no voters:
I20260925 07:20:47.681339 6585 raft_consensus.cc:2781] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:47.681406 6585 raft_consensus.cc:686] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 1 LEADER]: Becoming Leader. State: Replica: d890cde091734ee099eae0f5754d2b1f, State: Running, Role: LEADER
I20260925 07:20:47.681452 6585 consensus_queue.cc:237] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } }
I20260925 07:20:47.681996 6010 catalog_manager.cc:5495] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d890cde091734ee099eae0f5754d2b1f (127.4.47.2). New cstate: current_term: 1 leader_uuid: "d890cde091734ee099eae0f5754d2b1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d890cde091734ee099eae0f5754d2b1f" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 } health_report { overall_health: UNKNOWN } } }
I20260925 07:20:47.696522 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:47.694210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.4.47.2:40017)
0925 07:20:47.694374 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:47.694376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:47.694382 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:20:47.694403 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:47.694404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:47.694418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:47.694491 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:47.694595 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.695099 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.695808 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:47.696040 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:47.696112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:47.696252 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:47.696288 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:47.696353 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:47.696413 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:20:47.696435 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260925 07:20:47.698485 6449 raft_consensus.cc:1261] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Refusing update from remote peer d890cde091734ee099eae0f5754d2b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:47.698510 6169 raft_consensus.cc:1261] T 2634a7bdf1c04f5abbc307158f7ad1f8 P c84ccfd04de04240b37d4662c372b614 [term 1 FOLLOWER]: Refusing update from remote peer d890cde091734ee099eae0f5754d2b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:47.698693 6598 consensus_queue.cc:1035] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ccfd04de04240b37d4662c372b614" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:47.698809 6585 consensus_queue.cc:1035] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "296bd2e3490c4030a915f683a4db3e5f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 07:20:47.703220 6590 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 07:20:49.222389 6010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:49.222497 6010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:49.223256 6010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665a58bd05414508b1935397561e7a6c: Sending DeleteTablet for 3 replicas of tablet 2634a7bdf1c04f5abbc307158f7ad1f8
I20260925 07:20:49.223605 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:49 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260925 07:20:49.223635 6289 tablet_service.cc:1514] Processing DeleteTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:49 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260925 07:20:49.223670 6428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2634a7bdf1c04f5abbc307158f7ad1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:49 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44418
I20260925 07:20:49.223682 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6084
I20260925 07:20:49.225039 6666 tablet_replica.cc:330] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: stopping tablet replica
I20260925 07:20:49.225119 6666 raft_consensus.cc:2229] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:49.225160 6666 raft_consensus.cc:2258] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:49.225304 6665 tablet_replica.cc:330] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: stopping tablet replica
I20260925 07:20:49.225391 6665 raft_consensus.cc:2229] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:49.225436 6666 ts_tablet_manager.cc:1905] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:49.225468 6665 raft_consensus.cc:2258] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:49.225705 6665 ts_tablet_manager.cc:1905] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:49.227254 6666 ts_tablet_manager.cc:1918] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:20:49.227319 6666 log.cc:1196] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/2634a7bdf1c04f5abbc307158f7ad1f8
I20260925 07:20:49.227454 6665 ts_tablet_manager.cc:1918] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:20:49.227520 6665 log.cc:1196] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/2634a7bdf1c04f5abbc307158f7ad1f8
I20260925 07:20:49.227536 6666 ts_tablet_manager.cc:1939] T 2634a7bdf1c04f5abbc307158f7ad1f8 P 296bd2e3490c4030a915f683a4db3e5f: Deleting consensus metadata
I20260925 07:20:49.227783 6665 ts_tablet_manager.cc:1939] T 2634a7bdf1c04f5abbc307158f7ad1f8 P d890cde091734ee099eae0f5754d2b1f: Deleting consensus metadata
I20260925 07:20:49.227854 5996 catalog_manager.cc:4841] TS 296bd2e3490c4030a915f683a4db3e5f (127.4.47.3:42925): tablet 2634a7bdf1c04f5abbc307158f7ad1f8 (table test-table [id=1d5237951a9d4bb08f717719354801ee]) successfully deleted
I20260925 07:20:49.228098 5996 catalog_manager.cc:4841] TS d890cde091734ee099eae0f5754d2b1f (127.4.47.2:40017): tablet 2634a7bdf1c04f5abbc307158f7ad1f8 (table test-table [id=1d5237951a9d4bb08f717719354801ee]) successfully deleted
W20260925 07:20:49.231638 5996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:37407 (error 108)
I20260925 07:20:49.231930 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6224
I20260925 07:20:49.233572 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:49.232035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37407 (local address 127.0.0.1:48324)
0925 07:20:49.233488 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:49.233533 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:37407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1425,"thread_start_us":57,"threads_started":1}
W20260925 07:20:49.233718 5996 catalog_manager.cc:4584] TS c84ccfd04de04240b37d4662c372b614 (127.4.47.1:37407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2634a7bdf1c04f5abbc307158f7ad1f8: Network error: Client connection negotiation failed: client connection to 127.4.47.1:37407: connect: Connection refused (error 111)
I20260925 07:20:49.238039 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6364
I20260925 07:20:49.244093 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 5980
2026-09-25T07:20:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5852 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:49 dist-test-slave-zw07 krb5kdc[6671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:49 dist-test-slave-zw07 krb5kdc[6671](info): set up 2 sockets
Sep 25 07:20:49 dist-test-slave-zw07 krb5kdc[6671](info): commencing operation
krb5kdc: starting...
W20260925 07:20:50.306877 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:50.429844 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41993
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:41993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:50.509478 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:50.509729 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:50.509763 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:50.510898 6684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:50.510972 6684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:50.510995 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:50.511018 6684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:50.511045 6684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:50.512539 6684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:41993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:50.512817 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:50.513046 6684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:50.517997 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.518132 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.517990 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:50.518262 6684 server_base.cc:1034] running on GCE node
I20260925 07:20:50.518518 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:50.518795 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:50.519976 6684 hybrid_clock.cc:648] HybridClock initialized: now 1790320850519968 us; error 33 us; skew 500 ppm
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:50.522853 6684 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:50.523417 6684 webserver.cc:460] Webserver started at http://127.4.47.62:41477/ using document root <none> and password file <none>
I20260925 07:20:50.523692 6684 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:50.523766 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:50.523927 6684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:50.525107 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "a8d5c34bb797483696c62ad1697ee17e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "581308c1db997453c4b876a5392453fd"
server_key_iv: "1bc3a4f6b85d2227fc1c1ef39d056dff"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.525492 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "a8d5c34bb797483696c62ad1697ee17e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "581308c1db997453c4b876a5392453fd"
server_key_iv: "1bc3a4f6b85d2227fc1c1ef39d056dff"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.526911 6684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:50.527776 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.527942 6684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:50.528051 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "a8d5c34bb797483696c62ad1697ee17e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "581308c1db997453c4b876a5392453fd"
server_key_iv: "1bc3a4f6b85d2227fc1c1ef39d056dff"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.528136 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:50.564379 6684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:50.566977 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:50.567198 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:50.571518 6684 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:41993
I20260925 07:20:50.571553 6751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:41993 every 8 connection(s)
I20260925 07:20:50.571991 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:50.572928 6752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:50.574828 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6684
I20260925 07:20:50.574930 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:50.575126 4284 external_mini_cluster.cc:1428] Setting key 723922ebf1b35e79ee925c8f130e79d7
I20260925 07:20:50.575572 6752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e: Bootstrap starting.
I20260925 07:20:50.576217 6752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:50.576519 6752 log.cc:824] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:50.577287 6752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e: No bootstrap required, opened a new log
I20260925 07:20:50.579278 6752 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } }
I20260925 07:20:50.579450 6752 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:50.579483 6752 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d5c34bb797483696c62ad1697ee17e, State: Initialized, Role: FOLLOWER
I20260925 07:20:50.579617 6752 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } }
I20260925 07:20:50.579711 6752 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:50.579757 6752 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:50.579814 6752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:50.580406 6752 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } }
I20260925 07:20:50.580523 6752 leader_election.cc:302] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d5c34bb797483696c62ad1697ee17e; no voters:
I20260925 07:20:50.580678 6752 leader_election.cc:288] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:50.580734 6757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:50.580967 6757 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [term 1 LEADER]: Becoming Leader. State: Replica: a8d5c34bb797483696c62ad1697ee17e, State: Running, Role: LEADER
I20260925 07:20:50.581085 6757 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } }
I20260925 07:20:50.580974 6752 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:50.581431 6759 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d5c34bb797483696c62ad1697ee17e. Latest consensus state: current_term: 1 leader_uuid: "a8d5c34bb797483696c62ad1697ee17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } } }
I20260925 07:20:50.581589 6759 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:50.582162 6764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:50.582628 6764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:50.582741 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8d5c34bb797483696c62ad1697ee17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d5c34bb797483696c62ad1697ee17e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41993 } } }
I20260925 07:20:50.582835 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:50.584367 6764 catalog_manager.cc:1347] Generated new cluster ID: 80e55006da864c069550ae0d164dec0c
I20260925 07:20:50.584420 6764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:50.586858 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:50.575711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.4.47.62:41993)
0925 07:20:50.575890 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:50.575900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:50.575935 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:20:50.576491 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:50.576498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:50.576549 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:50.576720 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:50.577481 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.578555 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.579664 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.580204 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.580293 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:50.583305 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:50.583309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:50.583309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:50.583321 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:50.585273 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.585682 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.585684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.585685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.585783 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.585989 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.585989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.585989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.586259 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:50.586408 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:50.586521 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 07:20:50.586643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 07:20:50.609033 6764 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:50.609786 6764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:50.614140 6764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d5c34bb797483696c62ad1697ee17e: Generated new TSK 0
I20260925 07:20:50.614394 6764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:50.642926 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:50.736858 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:50.737082 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:50.737106 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:50.738328 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:50.738364 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:50.738392 6778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:50.739945 6778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:50.740154 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:50.740350 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:20:50.745771 6778 server_base.cc:1034] running on GCE node
W20260925 07:20:50.745738 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.745721 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.745738 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:50.746167 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:50.746418 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:50.747572 6778 hybrid_clock.cc:648] HybridClock initialized: now 1790320850747566 us; error 35 us; skew 500 ppm
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:50.750077 6778 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:50.750504 6778 webserver.cc:460] Webserver started at http://127.4.47.1:33717/ using document root <none> and password file <none>
I20260925 07:20:50.750721 6778 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:50.750792 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:50.750941 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:50.752074 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "968092e1b9ff4246967cdb78d824008b"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "513f654077019c64273adfad378e7a57"
server_key_iv: "eaed5bd6071f65e9be5a97514635fe11"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.752420 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "968092e1b9ff4246967cdb78d824008b"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "513f654077019c64273adfad378e7a57"
server_key_iv: "eaed5bd6071f65e9be5a97514635fe11"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.753752 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:20:50.754492 6793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.754645 6778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:50.754751 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "968092e1b9ff4246967cdb78d824008b"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "513f654077019c64273adfad378e7a57"
server_key_iv: "eaed5bd6071f65e9be5a97514635fe11"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.754804 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:50.767674 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:50.769912 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:50.770021 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:50.770195 6778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:50.770502 6778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:50.770535 6778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.770576 6778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:50.770601 6778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.776225 6778 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:45737
I20260925 07:20:50.776302 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:45737 every 8 connection(s)
I20260925 07:20:50.776605 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:50.778327 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6778
I20260925 07:20:50.778432 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:50.778635 4284 external_mini_cluster.cc:1428] Setting key 7b154f6a5d2bb64e0d10f5871da4507d
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:50.786660 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:50.777368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46587 (local address 127.4.47.62:41993)
0925 07:20:50.777430 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:50.777432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:50.777653 (+ 221us) server_negotiation.cc:407] Connection header received
0925 07:20:50.778148 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:50.778149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:50.778162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:50.778240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:50.779557 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.780052 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.781279 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.781571 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.781642 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:50.783928 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:50.783930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:50.783930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:50.783938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:50.785305 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.785686 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.785688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.785688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.785745 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.785986 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.785987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.785987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.786099 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:50.786175 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:50.786459 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 07:20:50.786550 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:20:50.786852 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:50.777354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41993 (local address 127.4.47.1:46587)
0925 07:20:50.777523 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:50.777565 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:50.778046 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:50.778337 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:50.778342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:50.779173 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:50.779472 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:50.779478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.780142 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.780144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:50.781198 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:50.781201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.781649 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.781651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:50.781818 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:50.782348 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:50.782356 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:50.783836 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:50.785388 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:50.785395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:50.785397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:50.785615 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:50.785809 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:50.785810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:50.785810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:50.785922 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:50.786165 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:50.786171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:50.786330 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 07:20:50.786438 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:20:50.786595 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
I20260925 07:20:50.787468 6907 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41993
I20260925 07:20:50.787559 6907 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:50.787755 6907 heartbeater.cc:511] Master 127.4.47.62:41993 requested a full tablet report, sending...
I20260925 07:20:50.788241 6716 ts_manager.cc:194] Registered new tserver with Master: 968092e1b9ff4246967cdb78d824008b (127.4.47.1:45737)
I20260925 07:20:50.788904 6716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:46587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:50.806802 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:50.888154 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:50.888357 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:50.888381 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:50.889559 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:50.889600 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:50.889626 6912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:50.891037 6912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:50.891304 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:50.891528 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:50.896281 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.896355 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:50.896274 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:50.896565 6912 server_base.cc:1034] running on GCE node
I20260925 07:20:50.896827 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:50.897051 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:50.898200 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790320850898189 us; error 38 us; skew 500 ppm
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:50.900609 6912 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:50.900969 6912 webserver.cc:460] Webserver started at http://127.4.47.2:40699/ using document root <none> and password file <none>
I20260925 07:20:50.901152 6912 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:50.901192 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:50.901283 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:50.902297 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "bf1880d677534f25b4e6ae2ef07be89e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "175edd5077906ba027f795132b284bdc"
server_key_iv: "012cee03f974416419cdddd87c6665e6"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.902590 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "bf1880d677534f25b4e6ae2ef07be89e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "175edd5077906ba027f795132b284bdc"
server_key_iv: "012cee03f974416419cdddd87c6665e6"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.903883 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:50.904606 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.904930 6912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:50.905021 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "bf1880d677534f25b4e6ae2ef07be89e"
format_stamp: "Formatted at 2026-09-25 07:20:50 on dist-test-slave-zw07"
server_key: "175edd5077906ba027f795132b284bdc"
server_key_iv: "012cee03f974416419cdddd87c6665e6"
server_key_version: "encryptionkey@0"
I20260925 07:20:50.905076 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:50.912081 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:50.913957 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:50.914064 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:50.914227 6912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:50.914455 6912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:50.914485 6912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.914528 6912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:50.914556 6912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:50.920410 6912 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:39161
I20260925 07:20:50.920490 7040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:39161 every 8 connection(s)
I20260925 07:20:50.920811 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:50.921907 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 6912
I20260925 07:20:50.922027 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:50.922254 4284 external_mini_cluster.cc:1428] Setting key 3d74f77a5dba418a0dddbf39010261f6
Sep 25 07:20:50 dist-test-slave-zw07 krb5kdc[6671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
I20260925 07:20:50.933462 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:50.921782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:48223 (local address 127.4.47.62:41993)
0925 07:20:50.921858 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:50.921862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:50.921967 (+ 105us) server_negotiation.cc:407] Connection header received
0925 07:20:50.922620 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:50.922621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:50.922634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:50.922724 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:50.923867 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.924429 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.925470 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.925808 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.925882 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:50.929885 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:50.929886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:50.929887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:50.929895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:50.932055 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.932491 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.932492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.932493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.932550 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:50.932806 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:50.932807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:50.932808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:50.932970 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:50.933049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:50.933317 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 07:20:50.933395 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:20:50.933676 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:50.921572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41993 (local address 127.4.47.2:48223)
0925 07:20:50.921834 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:50.921876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:50.922504 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:50.922816 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:50.922821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:50.923551 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:50.923754 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:50.923759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.924538 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.924539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:50.925382 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:50.925384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:50.925907 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:50.925908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:50.926018 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:50.926554 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:50.926562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:50.929780 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:50.932137 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:50.932143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:50.932145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:50.932408 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:50.932630 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:50.932630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:50.932630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:50.932738 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:50.933037 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:50.933041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:50.933186 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 07:20:50.933277 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 07:20:50.933416 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260925 07:20:50.934192 7041 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41993
I20260925 07:20:50.934306 7041 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:50.934485 7041 heartbeater.cc:511] Master 127.4.47.62:41993 requested a full tablet report, sending...
I20260925 07:20:50.934917 6716 ts_manager.cc:194] Registered new tserver with Master: bf1880d677534f25b4e6ae2ef07be89e (127.4.47.2:39161)
I20260925 07:20:50.935406 6716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:48223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:50.950024 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:51.031473 7046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:51.031689 7046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:51.031713 7046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:51.032919 7046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:51.032963 7046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:51.032989 7046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:51.034441 7046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:51.034634 7046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:51.034822 7046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:51.039824 7051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:51.039840 7046 server_base.cc:1034] running on GCE node
W20260925 07:20:51.039819 7054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:51.039812 7052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:51.040292 7046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:51.040511 7046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:51.041682 7046 hybrid_clock.cc:648] HybridClock initialized: now 1790320851041661 us; error 30 us; skew 500 ppm
Sep 25 07:20:51 dist-test-slave-zw07 krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:51.044243 7046 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:51.044718 7046 webserver.cc:460] Webserver started at http://127.4.47.3:38579/ using document root <none> and password file <none>
I20260925 07:20:51.044934 7046 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:51.044976 7046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:51.045066 7046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:51.046069 7046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "1f20b046005a4123a0c7fb80d06fde60"
format_stamp: "Formatted at 2026-09-25 07:20:51 on dist-test-slave-zw07"
server_key: "a3584d6854dbd60a72aa9a91c06069ce"
server_key_iv: "34c6e9927c26999996a3c7245651c29d"
server_key_version: "encryptionkey@0"
I20260925 07:20:51.046363 7046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "1f20b046005a4123a0c7fb80d06fde60"
format_stamp: "Formatted at 2026-09-25 07:20:51 on dist-test-slave-zw07"
server_key: "a3584d6854dbd60a72aa9a91c06069ce"
server_key_iv: "34c6e9927c26999996a3c7245651c29d"
server_key_version: "encryptionkey@0"
I20260925 07:20:51.047683 7046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:51.048367 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:51.048533 7046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:51.048616 7046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "1f20b046005a4123a0c7fb80d06fde60"
format_stamp: "Formatted at 2026-09-25 07:20:51 on dist-test-slave-zw07"
server_key: "a3584d6854dbd60a72aa9a91c06069ce"
server_key_iv: "34c6e9927c26999996a3c7245651c29d"
server_key_version: "encryptionkey@0"
I20260925 07:20:51.048672 7046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:51.061848 7046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:51.064103 7046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:51.064218 7046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:51.064412 7046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:51.064716 7046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:51.064757 7046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:51.064783 7046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:51.064826 7046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:51.070443 7046 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:44855
I20260925 07:20:51.070516 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:44855 every 8 connection(s)
I20260925 07:20:51.070816 7046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:20:51.075095 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7046
I20260925 07:20:51.075220 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:51.075433 4284 external_mini_cluster.cc:1428] Setting key 897267427ef1fc205880b0bbea4a43e4
Sep 25 07:20:51 dist-test-slave-zw07 krb5kdc[6671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:51.079873 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:51.071545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:35359 (local address 127.4.47.62:41993)
0925 07:20:51.071601 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:51.071603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:51.071799 (+ 196us) server_negotiation.cc:407] Connection header received
0925 07:20:51.072279 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:51.072280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:51.072289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:51.072349 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:51.073255 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:51.073756 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:51.074778 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:51.074957 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:51.075008 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:51.077330 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:51.077331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:51.077331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:51.077337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:51.078594 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:51.078965 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:51.078966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:51.078966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:51.079013 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:51.079286 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:51.079286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:51.079286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:51.079391 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:51.079445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:51.079718 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 07:20:51.079806 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:20:51.080016 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:51.071492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41993 (local address 127.4.47.3:35359)
0925 07:20:51.071662 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:51.071705 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:51.072189 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:51.072407 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:51.072410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:51.072980 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:51.073178 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:51.073183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:51.073854 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:51.073855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:51.074680 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:51.074682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:51.075052 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:51.075053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:51.075250 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:51.075854 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:51.075862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:51.077240 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:51.078667 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:51.078672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:51.078674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:51.078904 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:51.079105 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:51.079106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:51.079106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:51.079224 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:51.079450 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:51.079455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:51.079569 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 07:20:51.079657 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:20:51.079805 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260925 07:20:51.080480 7175 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41993
I20260925 07:20:51.080564 7175 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:51.080756 7175 heartbeater.cc:511] Master 127.4.47.62:41993 requested a full tablet report, sending...
I20260925 07:20:51.081153 6716 ts_manager.cc:194] Registered new tserver with Master: 1f20b046005a4123a0c7fb80d06fde60 (127.4.47.3:44855)
I20260925 07:20:51.081598 6716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:35359
I20260925 07:20:51.087124 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:20:51.097846 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:20:51.096935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.4.47.62:41993)
0925 07:20:51.097019 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:51.097022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:51.097087 (+ 65us) server_negotiation.cc:407] Connection header received
0925 07:20:51.097130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:51.097131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:51.097140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:51.097190 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:51.097770 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48414: BlockingRecv error: recv got EOF from 127.0.0.1:48414 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 07:20:51.098681 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6778
I20260925 07:20:51.103005 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6912
I20260925 07:20:51.107314 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7046
I20260925 07:20:51.111490 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 6684
2026-09-25T07:20:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1862 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:51 dist-test-slave-zw07 krb5kdc[7187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:51 dist-test-slave-zw07 krb5kdc[7187](info): set up 2 sockets
Sep 25 07:20:51 dist-test-slave-zw07 krb5kdc[7187](info): commencing operation
krb5kdc: starting...
W20260925 07:20:52.151315 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:52.270344 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33821
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:52.356930 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:52.357153 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:52.357178 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:52.358276 7200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:52.358312 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:52.358326 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:52.358338 7200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:52.358350 7200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:52.359913 7200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:52.360119 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:52.360308 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:52.365166 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:52.365151 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:52.365231 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:52.365563 7200 server_base.cc:1034] running on GCE node
I20260925 07:20:52.365834 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:52.366129 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:52.367270 7200 hybrid_clock.cc:648] HybridClock initialized: now 1790320852367258 us; error 42 us; skew 500 ppm
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320852, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:52.369650 7200 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:52.370113 7200 webserver.cc:460] Webserver started at http://127.4.47.62:46877/ using document root <none> and password file <none>
I20260925 07:20:52.370350 7200 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:52.370417 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:52.370569 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:52.371729 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "9e39228c794f4f8b962fdd1adf249f18"
format_stamp: "Formatted at 2026-09-25 07:20:52 on dist-test-slave-zw07"
server_key: "9ec3e16a739919b93eb6dc372f89768a"
server_key_iv: "a0cb3168448cb1186cef61f9d0e512c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:52.372097 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "9e39228c794f4f8b962fdd1adf249f18"
format_stamp: "Formatted at 2026-09-25 07:20:52 on dist-test-slave-zw07"
server_key: "9ec3e16a739919b93eb6dc372f89768a"
server_key_iv: "a0cb3168448cb1186cef61f9d0e512c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:52.373495 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:52.374251 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:52.374462 7200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:52.374570 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "9e39228c794f4f8b962fdd1adf249f18"
format_stamp: "Formatted at 2026-09-25 07:20:52 on dist-test-slave-zw07"
server_key: "9ec3e16a739919b93eb6dc372f89768a"
server_key_iv: "a0cb3168448cb1186cef61f9d0e512c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:52.374643 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:52.397917 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:52.400271 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:52.400450 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:52.404946 7200 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33821
I20260925 07:20:52.404978 7267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33821 every 8 connection(s)
I20260925 07:20:52.405354 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:52.406234 7268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:52.408649 7268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18: Bootstrap starting.
I20260925 07:20:52.409257 7268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:52.409557 7268 log.cc:824] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:52.410251 7268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18: No bootstrap required, opened a new log
I20260925 07:20:52.411882 7268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } }
I20260925 07:20:52.412024 7268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:52.412046 7268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39228c794f4f8b962fdd1adf249f18, State: Initialized, Role: FOLLOWER
I20260925 07:20:52.412191 7268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } }
I20260925 07:20:52.412261 7268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:52.412295 7268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:52.412348 7268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:52.413468 7268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } }
I20260925 07:20:52.413565 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e39228c794f4f8b962fdd1adf249f18; no voters:
I20260925 07:20:52.413791 7268 leader_election.cc:288] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:52.413923 7271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:52.414093 7271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [term 1 LEADER]: Becoming Leader. State: Replica: 9e39228c794f4f8b962fdd1adf249f18, State: Running, Role: LEADER
I20260925 07:20:52.414172 7268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:52.414186 7271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } }
I20260925 07:20:52.414652 7273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e39228c794f4f8b962fdd1adf249f18. Latest consensus state: current_term: 1 leader_uuid: "9e39228c794f4f8b962fdd1adf249f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } } }
I20260925 07:20:52.414662 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e39228c794f4f8b962fdd1adf249f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e39228c794f4f8b962fdd1adf249f18" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33821 } } }
I20260925 07:20:52.414743 7273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:52.414775 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:52.415216 7280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:52.415648 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7200
I20260925 07:20:52.415752 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:52.415763 7280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:52.415964 4284 external_mini_cluster.cc:1428] Setting key b4e9cb4059b33393149cf61d05a35ca0
I20260925 07:20:52.417531 7280 catalog_manager.cc:1347] Generated new cluster ID: a5f4583192bd4bedbc411cfc9a535ec2
I20260925 07:20:52.417583 7280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:52.435111 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:52.416442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.4.47.62:33821)
0925 07:20:52.417003 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:52.417013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:52.423398 (+ 6385us) server_negotiation.cc:407] Connection header received
0925 07:20:52.423550 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:52.423558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:52.423622 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:52.423787 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:52.424621 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:52.426151 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:52.427518 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:52.428233 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:52.428345 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:52.431310 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:52.431315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:52.431316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:52.431330 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:52.433264 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:52.433712 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:52.433716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:52.433717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:52.433841 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:52.434055 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:52.434055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:52.434056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:52.434375 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:52.434511 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:52.434633 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 07:20:52.434849 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":81,"threads_started":1}
I20260925 07:20:52.436036 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:20:52.437312 7280 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:52.438020 7280 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:52.449111 7280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e39228c794f4f8b962fdd1adf249f18: Generated new TSK 0
I20260925 07:20:52.449331 7280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:20:52.458124 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:52.454029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.4.47.62:33821)
0925 07:20:52.454083 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:52.454085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:52.454183 (+ 98us) server_negotiation.cc:407] Connection header received
0925 07:20:52.454279 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:52.454280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:52.454290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:52.454352 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:52.454826 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:52.455670 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:52.456423 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:52.456758 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:52.456833 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:52.457569 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:52.457570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:52.457571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:52.457850 (+ 279us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:52.457869 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:52.457876 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:52.457939 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:52.458011 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 07:20:52.458040 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:20:52.460744 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7200
2026-09-25T07:20:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1347 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7301](info): set up 2 sockets
Sep 25 07:20:52 dist-test-slave-zw07 krb5kdc[7301](info): commencing operation
krb5kdc: starting...
W20260925 07:20:53.519261 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:53 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:53.638257 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38679
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:53.718899 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:53.719192 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:53.719223 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:53.720408 7314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:53.720449 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:53.720465 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:53.720477 7314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:53.720489 7314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:53.721977 7314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:53.722229 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:53.722477 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:53.727331 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:53.727349 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:53.727350 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:53.727561 7314 server_base.cc:1034] running on GCE node
I20260925 07:20:53.727778 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:53.728026 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:53.729214 7314 hybrid_clock.cc:648] HybridClock initialized: now 1790320853729198 us; error 29 us; skew 500 ppm
Sep 25 07:20:53 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:53.731757 7314 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:53.732278 7314 webserver.cc:460] Webserver started at http://127.4.47.62:34621/ using document root <none> and password file <none>
I20260925 07:20:53.732525 7314 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:53.732570 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:53.732724 7314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:53.733791 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "d670cd40e1034fa8bf7e7207c167a8d7"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "9d51765f630bee622665e124b3cc9253"
server_key_iv: "864ddaea29da9b15a77a1f814c3420b6"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.734154 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "d670cd40e1034fa8bf7e7207c167a8d7"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "9d51765f630bee622665e124b3cc9253"
server_key_iv: "864ddaea29da9b15a77a1f814c3420b6"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.735571 7314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:53.736366 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:53.736542 7314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:53.736634 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "d670cd40e1034fa8bf7e7207c167a8d7"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "9d51765f630bee622665e124b3cc9253"
server_key_iv: "864ddaea29da9b15a77a1f814c3420b6"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.736711 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:53.753469 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:53.755436 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:53.755596 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:53.759665 7314 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:38679
I20260925 07:20:53.759701 7381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:38679 every 8 connection(s)
I20260925 07:20:53.760051 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:53.761018 7382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:53.762941 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7314
I20260925 07:20:53.763103 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:53.763279 4284 external_mini_cluster.cc:1428] Setting key b77b5c754921c4480c4fcb0e99e6b879
I20260925 07:20:53.763469 7382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7: Bootstrap starting.
I20260925 07:20:53.764149 7382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:53.764458 7382 log.cc:824] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:53.765226 7382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7: No bootstrap required, opened a new log
I20260925 07:20:53.767532 7382 raft_consensus.cc:348] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } }
I20260925 07:20:53.767714 7382 raft_consensus.cc:374] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:53.767742 7382 raft_consensus.cc:729] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d670cd40e1034fa8bf7e7207c167a8d7, State: Initialized, Role: FOLLOWER
I20260925 07:20:53.767889 7382 consensus_queue.cc:260] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } }
I20260925 07:20:53.767969 7382 raft_consensus.cc:388] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:53.768011 7382 raft_consensus.cc:482] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:53.768060 7382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:53.768774 7382 raft_consensus.cc:504] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } }
I20260925 07:20:53.768889 7382 leader_election.cc:302] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d670cd40e1034fa8bf7e7207c167a8d7; no voters:
I20260925 07:20:53.769060 7382 leader_election.cc:288] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:53.769086 7387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:53.769254 7387 raft_consensus.cc:686] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [term 1 LEADER]: Becoming Leader. State: Replica: d670cd40e1034fa8bf7e7207c167a8d7, State: Running, Role: LEADER
I20260925 07:20:53.769331 7382 sys_catalog.cc:564] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:53.769349 7387 consensus_queue.cc:237] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } }
I20260925 07:20:53.769630 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d670cd40e1034fa8bf7e7207c167a8d7. Latest consensus state: current_term: 1 leader_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } } }
I20260925 07:20:53.769727 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:53.769644 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d670cd40e1034fa8bf7e7207c167a8d7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38679 } } }
I20260925 07:20:53.769848 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:53.769932 7392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:53.770432 7392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:20:53 dist-test-slave-zw07 krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:53.772270 7392 catalog_manager.cc:1347] Generated new cluster ID: 8f0fe027f875469a91d8c587de7c8f55
I20260925 07:20:53.772320 7392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:53.777724 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:53.763823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.4.47.62:38679)
0925 07:20:53.764018 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:53.764028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:53.764062 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:20:53.764566 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:53.764575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:53.764652 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:53.764816 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:53.765573 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.766449 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.767624 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.768245 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.768327 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:53.771194 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:53.771199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:53.771199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:53.771218 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:53.776065 (+ 4847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:53.776491 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:53.776494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:53.776494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:53.776609 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:53.776856 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:53.776857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:53.776857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:53.777137 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:53.777278 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:53.777409 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 07:20:53.777538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260925 07:20:53.798523 7392 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:53.799332 7392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:53.806646 7392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d670cd40e1034fa8bf7e7207c167a8d7: Generated new TSK 0
I20260925 07:20:53.806849 7392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:53.840276 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:53.921397 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:53.921604 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:53.921627 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:53.922724 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:53.922758 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:53.922783 7408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:53.924207 7408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:53.924386 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:53.924614 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:53.929533 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:53.929526 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:53.929526 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:53.929778 7408 server_base.cc:1034] running on GCE node
I20260925 07:20:53.930120 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:53.930405 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:53.931574 7408 hybrid_clock.cc:648] HybridClock initialized: now 1790320853931565 us; error 37 us; skew 500 ppm
Sep 25 07:20:53 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:53.934120 7408 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:53.934584 7408 webserver.cc:460] Webserver started at http://127.4.47.1:38461/ using document root <none> and password file <none>
I20260925 07:20:53.934831 7408 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:53.934898 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:53.935040 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:53.936139 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "ad7e3e93bac0424698f954bb6817c41f"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "464bf4f0bad5167e95a81822857d8967"
server_key_iv: "fbd15fe09b73b44d6c16f69babc1db51"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.936506 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "ad7e3e93bac0424698f954bb6817c41f"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "464bf4f0bad5167e95a81822857d8967"
server_key_iv: "fbd15fe09b73b44d6c16f69babc1db51"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.937848 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:20:53.938614 7423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:53.938796 7408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:53.938892 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "ad7e3e93bac0424698f954bb6817c41f"
format_stamp: "Formatted at 2026-09-25 07:20:53 on dist-test-slave-zw07"
server_key: "464bf4f0bad5167e95a81822857d8967"
server_key_iv: "fbd15fe09b73b44d6c16f69babc1db51"
server_key_version: "encryptionkey@0"
I20260925 07:20:53.938971 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:53.968194 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:53.970459 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:53.970633 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:53.970881 7408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:53.971216 7408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:53.971269 7408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:53.971304 7408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:53.971331 7408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:53.977053 7408 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:34461
I20260925 07:20:53.977100 7536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:34461 every 8 connection(s)
I20260925 07:20:53.977458 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:20:53 dist-test-slave-zw07 krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:53.985416 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7408
I20260925 07:20:53.985525 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:53.985733 4284 external_mini_cluster.cc:1428] Setting key 6c61deda90ff3c54bf823208af57a34d
I20260925 07:20:53.986825 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:53.978251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52479 (local address 127.4.47.62:38679)
0925 07:20:53.978336 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:53.978338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:53.978493 (+ 155us) server_negotiation.cc:407] Connection header received
0925 07:20:53.979002 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:53.979003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:53.979012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:53.979080 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:53.980025 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.980521 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.981552 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.981745 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.981797 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:53.983843 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:53.983845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:53.983846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:53.983865 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:53.985260 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:53.985740 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:53.985742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:53.985742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:53.985804 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:53.986131 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:53.986133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:53.986133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:53.986262 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:53.986326 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:53.986673 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 07:20:53.986743 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 07:20:53.987178 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:53.978209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38679 (local address 127.4.47.1:52479)
0925 07:20:53.978371 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:53.978414 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:53.978914 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:53.979193 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:53.979197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:53.979753 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:53.979943 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:53.979948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.980600 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.980601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:53.981474 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:53.981477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:53.981813 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:53.981813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:53.981910 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:53.982349 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:53.982357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:53.983725 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:53.985366 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:53.985373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:53.985375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:53.985653 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:53.985892 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:53.985893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:53.985893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:53.986050 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:53.986326 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:53.986331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:53.986529 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 07:20:53.986654 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 07:20:53.986860 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260925 07:20:53.987675 7537 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38679
I20260925 07:20:53.987771 7537 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:53.987954 7537 heartbeater.cc:511] Master 127.4.47.62:38679 requested a full tablet report, sending...
I20260925 07:20:53.988484 7346 ts_manager.cc:194] Registered new tserver with Master: ad7e3e93bac0424698f954bb6817c41f (127.4.47.1:34461)
I20260925 07:20:53.989077 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:52479
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:54.011150 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:54.092777 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:54.093020 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:54.093057 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:54.094213 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:54.094295 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:54.094342 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:54.095784 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:54.096047 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:54.096283 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:54.101303 7548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:54.101397 7542 server_base.cc:1034] running on GCE node
W20260925 07:20:54.101271 7550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:54.101303 7547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:54.101697 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:54.101950 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:54.103138 7542 hybrid_clock.cc:648] HybridClock initialized: now 1790320854103128 us; error 38 us; skew 500 ppm
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:54.105525 7542 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:54.106003 7542 webserver.cc:460] Webserver started at http://127.4.47.2:36251/ using document root <none> and password file <none>
I20260925 07:20:54.106228 7542 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:54.106307 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:54.106444 7542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:54.107564 7542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "9b32c5826cd642dd9c27b35f389dc6cb"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "23c101c0fe8fd9790eaef8e03cb8f26e"
server_key_iv: "6fe5ef3c02dd62729909a67c004552d6"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.107939 7542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "9b32c5826cd642dd9c27b35f389dc6cb"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "23c101c0fe8fd9790eaef8e03cb8f26e"
server_key_iv: "6fe5ef3c02dd62729909a67c004552d6"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.109306 7542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:54.110075 7557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.110272 7542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:54.110373 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "9b32c5826cd642dd9c27b35f389dc6cb"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "23c101c0fe8fd9790eaef8e03cb8f26e"
server_key_iv: "6fe5ef3c02dd62729909a67c004552d6"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.110457 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:54.131415 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:54.133496 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:54.133675 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:54.133946 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:54.134268 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:54.134322 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.134364 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:54.134411 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.140450 7542 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:42747
I20260925 07:20:54.140492 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:42747 every 8 connection(s)
I20260925 07:20:54.140820 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:54.145959 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7542
I20260925 07:20:54.146081 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:54.146299 4284 external_mini_cluster.cc:1428] Setting key 09eb2bead4a5f3532484d2ca1692d844
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:54.150261 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:54.141561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:34409 (local address 127.4.47.62:38679)
0925 07:20:54.141633 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:54.141635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:54.141870 (+ 235us) server_negotiation.cc:407] Connection header received
0925 07:20:54.142339 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:54.142340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:54.142349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:54.142408 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:54.143390 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.143890 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.144912 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.145086 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.145136 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:54.147406 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:54.147407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:54.147407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:54.147416 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:54.148861 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.149293 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.149295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.149295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.149355 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.149606 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.149607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.149608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.149730 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:54.149804 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:54.150079 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 07:20:54.150170 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 07:20:54.150467 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:54.141545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38679 (local address 127.4.47.2:34409)
0925 07:20:54.141696 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:54.141746 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:54.142253 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:54.142463 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:54.142466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:54.143100 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:54.143309 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:54.143314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.143975 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.143976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:54.144836 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:54.144838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.145156 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.145157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:54.145272 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:54.145730 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:54.145737 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:54.147312 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:54.148962 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:54.148969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:54.148971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:54.149194 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:54.149421 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:54.149422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:54.149423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:54.149538 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:54.149804 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:54.149811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:54.149950 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 07:20:54.150058 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:20:54.150226 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260925 07:20:54.150959 7671 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38679
I20260925 07:20:54.151052 7671 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:54.151271 7671 heartbeater.cc:511] Master 127.4.47.62:38679 requested a full tablet report, sending...
I20260925 07:20:54.151688 7346 ts_manager.cc:194] Registered new tserver with Master: 9b32c5826cd642dd9c27b35f389dc6cb (127.4.47.2:42747)
I20260925 07:20:54.152078 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:34409
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:54.172155 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:54.257103 7676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:54.257423 7676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:54.257467 7676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:54.259667 7676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:54.259720 7676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:54.259749 7676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:54.261842 7676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:54.262099 7676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:54.262354 7676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:54.267431 7682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:54.267557 7681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:54.267441 7684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:54.267619 7676 server_base.cc:1034] running on GCE node
I20260925 07:20:54.267911 7676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:54.268170 7676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:54.269346 7676 hybrid_clock.cc:648] HybridClock initialized: now 1790320854269331 us; error 24 us; skew 500 ppm
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:54.272130 7676 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:54.272614 7676 webserver.cc:460] Webserver started at http://127.4.47.3:45503/ using document root <none> and password file <none>
I20260925 07:20:54.272848 7676 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:54.272917 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:54.273058 7676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:54.274102 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "25f2cb33499b498699a927f119e78b47"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "87b1a6d32c378eeec0e1173606e5d60b"
server_key_iv: "91d331cefeac8527eb7b8e6d0cfa03cb"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.274466 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "25f2cb33499b498699a927f119e78b47"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "87b1a6d32c378eeec0e1173606e5d60b"
server_key_iv: "91d331cefeac8527eb7b8e6d0cfa03cb"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.275856 7676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:20:54.276588 7691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.276759 7676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:54.276855 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "25f2cb33499b498699a927f119e78b47"
format_stamp: "Formatted at 2026-09-25 07:20:54 on dist-test-slave-zw07"
server_key: "87b1a6d32c378eeec0e1173606e5d60b"
server_key_iv: "91d331cefeac8527eb7b8e6d0cfa03cb"
server_key_version: "encryptionkey@0"
I20260925 07:20:54.276934 7676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:54.289920 7676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:54.292032 7676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:54.292197 7676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:54.292441 7676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:54.292768 7676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:54.292829 7676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.292874 7676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:54.292898 7676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:54.298812 7676 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:42275
I20260925 07:20:54.298848 7804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:42275 every 8 connection(s)
I20260925 07:20:54.299228 7676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:20:54.307483 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7676
I20260925 07:20:54.307587 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:54.307760 4284 external_mini_cluster.cc:1428] Setting key ad9b8cf9061da4c4eacb3d1c2ccffc21
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:54.312602 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:54.299956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:57195 (local address 127.4.47.62:38679)
0925 07:20:54.300016 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:54.300018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:54.300217 (+ 199us) server_negotiation.cc:407] Connection header received
0925 07:20:54.300700 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:54.300701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:54.300710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:54.300764 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:54.301686 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.302169 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.303254 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.303451 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.303518 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:54.310049 (+ 6531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:54.310050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:54.310051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:54.310057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:54.311396 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.311824 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.311825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.311825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.311876 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.312090 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.312091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.312091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.312192 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:54.312242 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:54.312477 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 07:20:54.312538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:20:54.312858 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:54.299928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38679 (local address 127.4.47.3:57195)
0925 07:20:54.300084 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:54.300127 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:54.300619 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:54.300830 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:54.300833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:54.301411 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:54.301610 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:54.301614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.302248 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.302248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:54.303151 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:54.303153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.303533 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.303534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:54.303635 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:54.308556 (+ 4921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:54.308565 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:54.309949 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:54.311487 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:54.311493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:54.311495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:54.311753 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:54.311936 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:54.311936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:54.311937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:54.312036 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:54.312246 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:54.312250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:54.312368 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 07:20:54.312452 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:20:54.312572 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260925 07:20:54.313443 7805 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38679
I20260925 07:20:54.313553 7805 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:54.313741 7805 heartbeater.cc:511] Master 127.4.47.62:38679 requested a full tablet report, sending...
I20260925 07:20:54.314100 7346 ts_manager.cc:194] Registered new tserver with Master: 25f2cb33499b498699a927f119e78b47 (127.4.47.3:42275)
I20260925 07:20:54.314471 7346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:57195
I20260925 07:20:54.319738 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320854, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:54.343739 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:54.336477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.4.47.62:38679)
0925 07:20:54.336564 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:54.336566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:54.336665 (+ 99us) server_negotiation.cc:407] Connection header received
0925 07:20:54.336746 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:54.336747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:54.336757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:54.336808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:54.337393 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.337895 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.338640 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:54.338849 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:54.338937 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:54.341246 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:54.341248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:54.341248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:54.341257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:54.342546 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.342875 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.342876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.342876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.342936 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:54.343164 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:54.343165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:54.343165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:54.343329 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:54.343441 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:54.343545 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:20:54.343629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 07:20:54.343990 7346 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58992
I20260925 07:20:54.344429 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7408
I20260925 07:20:54.349242 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7542
I20260925 07:20:54.354288 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7676
I20260925 07:20:54.358688 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7314
2026-09-25T07:20:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1899 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7817](info): set up 2 sockets
Sep 25 07:20:54 dist-test-slave-zw07 krb5kdc[7817](info): commencing operation
krb5kdc: starting...
W20260925 07:20:55.383199 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:55.500303 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33307
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:55.581341 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:55.581575 7830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:55.581614 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:55.582762 7830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:55.582828 7830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:55.582852 7830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:55.582875 7830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:55.582906 7830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:55.584455 7830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:55.584724 7830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:55.584947 7830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:55.589818 7838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:55.589804 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.589869 7830 server_base.cc:1034] running on GCE node
W20260925 07:20:55.589792 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.590329 7830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:55.590603 7830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:55.591759 7830 hybrid_clock.cc:648] HybridClock initialized: now 1790320855591751 us; error 36 us; skew 500 ppm
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:55.594206 7830 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:55.594659 7830 webserver.cc:460] Webserver started at http://127.4.47.62:34575/ using document root <none> and password file <none>
I20260925 07:20:55.594844 7830 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:55.594885 7830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:55.594978 7830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:55.596069 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "3216ecc05b874e098753b966e24c5e78"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "151ae0ce5f915cfd4478c41c6d76f808"
server_key_iv: "bd4f80820ece8b98b3333134160e33c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.596383 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "3216ecc05b874e098753b966e24c5e78"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "151ae0ce5f915cfd4478c41c6d76f808"
server_key_iv: "bd4f80820ece8b98b3333134160e33c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.597662 7830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:55.598359 7845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.598537 7830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:55.598625 7830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "3216ecc05b874e098753b966e24c5e78"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "151ae0ce5f915cfd4478c41c6d76f808"
server_key_iv: "bd4f80820ece8b98b3333134160e33c5"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.598685 7830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:55.607766 7830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:55.609519 7830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:55.609632 7830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:55.613431 7830 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33307
I20260925 07:20:55.613497 7897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33307 every 8 connection(s)
I20260925 07:20:55.614004 7830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:55.614662 7898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:55.614953 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7830
I20260925 07:20:55.615058 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:55.615326 4284 external_mini_cluster.cc:1428] Setting key 3f30cae475bb76d76e52ee36475cd222
I20260925 07:20:55.617280 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78: Bootstrap starting.
I20260925 07:20:55.617861 7898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:55.618142 7898 log.cc:824] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:55.618777 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78: No bootstrap required, opened a new log
I20260925 07:20:55.620781 7898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } }
I20260925 07:20:55.620940 7898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:55.620965 7898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3216ecc05b874e098753b966e24c5e78, State: Initialized, Role: FOLLOWER
I20260925 07:20:55.621094 7898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } }
I20260925 07:20:55.621186 7898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:55.621209 7898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:55.621270 7898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:55.621809 7898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } }
I20260925 07:20:55.621892 7898 leader_election.cc:302] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3216ecc05b874e098753b966e24c5e78; no voters:
I20260925 07:20:55.622102 7898 leader_election.cc:288] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:55.622128 7903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:55.622243 7903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [term 1 LEADER]: Becoming Leader. State: Replica: 3216ecc05b874e098753b966e24c5e78, State: Running, Role: LEADER
I20260925 07:20:55.622354 7898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:55.622335 7903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } }
I20260925 07:20:55.622954 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3216ecc05b874e098753b966e24c5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } } }
I20260925 07:20:55.623046 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:55.623198 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3216ecc05b874e098753b966e24c5e78. Latest consensus state: current_term: 1 leader_uuid: "3216ecc05b874e098753b966e24c5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3216ecc05b874e098753b966e24c5e78" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33307 } } }
I20260925 07:20:55.623246 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:55.624053 7912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:55.624497 7912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:55.625764 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:55.615833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.4.47.62:33307)
0925 07:20:55.616044 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:55.616052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:55.616077 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:20:55.616481 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:55.616487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:55.616532 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:55.616689 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:55.617313 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.618332 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.619175 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.619579 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.619703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:55.622135 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:55.622140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:55.622140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:55.622152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:55.624251 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.624633 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.624636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.624637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.624727 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.624938 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.624939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.624940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.625180 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:55.625358 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:55.625427 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 07:20:55.625563 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260925 07:20:55.625964 7912 catalog_manager.cc:1347] Generated new cluster ID: 40eecc0b4b174535aa235f74aaa1b9b7
I20260925 07:20:55.625998 7912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:55.646540 7912 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:55.647356 7912 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:55.652282 7912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3216ecc05b874e098753b966e24c5e78: Generated new TSK 0
I20260925 07:20:55.652482 7912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:55.682804 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:55.764189 7924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:55.764394 7924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:55.764417 7924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:55.765584 7924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:55.765625 7924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:55.765681 7924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:55.767342 7924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:55.767596 7924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:55.767829 7924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:55.772957 7929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.772981 7924 server_base.cc:1034] running on GCE node
W20260925 07:20:55.772953 7930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:55.772933 7932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.773345 7924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:55.773552 7924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:55.774688 7924 hybrid_clock.cc:648] HybridClock initialized: now 1790320855774671 us; error 37 us; skew 500 ppm
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:55.777210 7924 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:55.777638 7924 webserver.cc:460] Webserver started at http://127.4.47.1:46203/ using document root <none> and password file <none>
I20260925 07:20:55.777873 7924 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:55.777920 7924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:55.778080 7924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:55.779192 7924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "e198398ef20a44098815213fa01bc487"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "59a9b73902e82695bf314a50a0620ad4"
server_key_iv: "97551c43f3e43d8abaf7ee50efad47f7"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.779537 7924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "e198398ef20a44098815213fa01bc487"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "59a9b73902e82695bf314a50a0620ad4"
server_key_iv: "97551c43f3e43d8abaf7ee50efad47f7"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.780807 7924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:55.781510 7939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.781661 7924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:55.781750 7924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "e198398ef20a44098815213fa01bc487"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "59a9b73902e82695bf314a50a0620ad4"
server_key_iv: "97551c43f3e43d8abaf7ee50efad47f7"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.781811 7924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:55.810751 7924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:55.812665 7924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:55.812780 7924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:55.812955 7924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:55.813288 7924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:55.813331 7924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.813354 7924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:55.813402 7924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.819267 7924 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:36365
I20260925 07:20:55.819355 8052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:36365 every 8 connection(s)
I20260925 07:20:55.819671 7924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:55.827898 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 7924
I20260925 07:20:55.828007 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:55.828217 4284 external_mini_cluster.cc:1428] Setting key 73839d1328c20cbf951b607a8a4820fe
I20260925 07:20:55.829887 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:55.820416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33307 (local address 127.4.47.1:33705)
0925 07:20:55.820559 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:55.820607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:55.821589 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:55.821871 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:55.821879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:55.822140 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:55.822385 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:55.822391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.823149 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.823150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:55.824449 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:55.824452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.824860 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.824861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:55.824964 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:55.825510 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:55.825519 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:55.826860 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:55.828480 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:55.828487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:55.828489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:55.828775 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:55.829002 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:55.829002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:55.829003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:55.829111 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:55.829393 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:55.829397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:55.829513 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 07:20:55.829596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:20:55.829718 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260925 07:20:55.829880 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:55.820456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:33705 (local address 127.4.47.62:33307)
0925 07:20:55.820502 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:55.820504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:55.820713 (+ 209us) server_negotiation.cc:407] Connection header received
0925 07:20:55.821707 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:55.821709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:55.821721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:55.821774 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:55.822487 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.823019 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.824550 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.824771 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.824836 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:55.826950 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:55.826951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:55.826951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:55.826958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:55.828396 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.828851 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.828852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.828852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.828910 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.829183 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.829184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.829184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.829328 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:55.829400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:55.829637 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 07:20:55.829733 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:20:55.830399 8053 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33307
I20260925 07:20:55.830511 8053 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:55.830686 8053 heartbeater.cc:511] Master 127.4.47.62:33307 requested a full tablet report, sending...
I20260925 07:20:55.831285 7862 ts_manager.cc:194] Registered new tserver with Master: e198398ef20a44098815213fa01bc487 (127.4.47.1:36365)
I20260925 07:20:55.831903 7862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:33705
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:55.854123 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:55.935016 8058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:55.935251 8058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:55.935274 8058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:55.936514 8058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:55.936566 8058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:55.936592 8058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:55.938117 8058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:55.938383 8058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:55.938616 8058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:55.943735 8063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.943782 8058 server_base.cc:1034] running on GCE node
W20260925 07:20:55.943732 8064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:55.943732 8066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:55.944111 8058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:55.944375 8058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:55.945536 8058 hybrid_clock.cc:648] HybridClock initialized: now 1790320855945524 us; error 29 us; skew 500 ppm
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:55.947880 8058 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:55.948354 8058 webserver.cc:460] Webserver started at http://127.4.47.2:46129/ using document root <none> and password file <none>
I20260925 07:20:55.948617 8058 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:55.948681 8058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:55.948822 8058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:55.949849 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "388a38bab52a4e5993e5ab5450309e62"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "ea0f7fecb3aa95da3b6b01176b14318a"
server_key_iv: "ec57f9ce0f65f816da18bd0175437770"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.950210 8058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "388a38bab52a4e5993e5ab5450309e62"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "ea0f7fecb3aa95da3b6b01176b14318a"
server_key_iv: "ec57f9ce0f65f816da18bd0175437770"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.951579 8058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:20:55.952322 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.952515 8058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:55.952611 8058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "388a38bab52a4e5993e5ab5450309e62"
format_stamp: "Formatted at 2026-09-25 07:20:55 on dist-test-slave-zw07"
server_key: "ea0f7fecb3aa95da3b6b01176b14318a"
server_key_iv: "ec57f9ce0f65f816da18bd0175437770"
server_key_version: "encryptionkey@0"
I20260925 07:20:55.952690 8058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:55.973383 8058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:55.975240 8058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:55.975409 8058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:55.975639 8058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:55.975946 8058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:55.975999 8058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.976040 8058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:55.976068 8058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:55.982029 8058 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:41111
I20260925 07:20:55.982079 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:41111 every 8 connection(s)
I20260925 07:20:55.982389 8058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:20:55 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:55.989197 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8058
I20260925 07:20:55.989295 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:55.989535 4284 external_mini_cluster.cc:1428] Setting key c02555c69980bff011412b3d413e1ba0
I20260925 07:20:55.992425 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:55.983097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33307 (local address 127.4.47.2:43937)
0925 07:20:55.983270 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:55.983327 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:55.984268 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:55.984518 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:55.984521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:55.984795 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:55.984992 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:55.984996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.985721 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.985722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:55.986557 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:55.986560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.986975 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.986976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:55.987150 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:55.987632 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:55.987640 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:55.989079 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:55.990859 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:55.990866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:55.990868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:55.991229 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:55.991462 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:55.991463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:55.991463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:55.991605 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:55.991931 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:55.991936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:55.992054 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 07:20:55.992131 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:20:55.992253 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 07:20:55.992429 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:55.983143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:43937 (local address 127.4.47.62:33307)
0925 07:20:55.983218 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:55.983220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:55.983407 (+ 187us) server_negotiation.cc:407] Connection header received
0925 07:20:55.984376 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:55.984377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:55.984390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:55.984443 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:55.985083 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.985631 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.986658 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:55.986877 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:55.986964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:55.989196 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:55.989197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:55.989198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:55.989206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:55.990769 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.991319 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.991320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.991320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.991388 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:55.991687 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:55.991688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:55.991689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:55.991863 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:55.991936 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:55.992194 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 07:20:55.992289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:20:55.992985 8187 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33307
I20260925 07:20:55.993076 8187 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:55.993270 8187 heartbeater.cc:511] Master 127.4.47.62:33307 requested a full tablet report, sending...
I20260925 07:20:55.993629 7862 ts_manager.cc:194] Registered new tserver with Master: 388a38bab52a4e5993e5ab5450309e62 (127.4.47.2:41111)
I20260925 07:20:55.994000 7862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:43937
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:56.013965 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:56.094954 8192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:56.095230 8192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:56.095280 8192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:56.096489 8192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:56.096570 8192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:56.096612 8192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:56.098138 8192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:56.098519 8192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:56.098769 8192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:56.103911 8200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:56.103907 8197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:56.103993 8198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:56.104053 8192 server_base.cc:1034] running on GCE node
I20260925 07:20:56.104379 8192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:56.104627 8192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:56.105782 8192 hybrid_clock.cc:648] HybridClock initialized: now 1790320856105770 us; error 28 us; skew 500 ppm
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:56.108351 8192 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:56.108847 8192 webserver.cc:460] Webserver started at http://127.4.47.3:45891/ using document root <none> and password file <none>
I20260925 07:20:56.109076 8192 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:56.109143 8192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:56.109303 8192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:56.110402 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "dbfb927d085d4841b9546e9ddc45fb91"
format_stamp: "Formatted at 2026-09-25 07:20:56 on dist-test-slave-zw07"
server_key: "432c8aac78000b91c936c3d20897fd2b"
server_key_iv: "62d7bd0250126c5358d6b92bb8eaa325"
server_key_version: "encryptionkey@0"
I20260925 07:20:56.110782 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "dbfb927d085d4841b9546e9ddc45fb91"
format_stamp: "Formatted at 2026-09-25 07:20:56 on dist-test-slave-zw07"
server_key: "432c8aac78000b91c936c3d20897fd2b"
server_key_iv: "62d7bd0250126c5358d6b92bb8eaa325"
server_key_version: "encryptionkey@0"
I20260925 07:20:56.112200 8192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:56.112916 8207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:56.113080 8192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:56.113178 8192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "dbfb927d085d4841b9546e9ddc45fb91"
format_stamp: "Formatted at 2026-09-25 07:20:56 on dist-test-slave-zw07"
server_key: "432c8aac78000b91c936c3d20897fd2b"
server_key_iv: "62d7bd0250126c5358d6b92bb8eaa325"
server_key_version: "encryptionkey@0"
I20260925 07:20:56.113296 8192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:56.122804 8192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:56.125217 8192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:56.125396 8192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:56.125650 8192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:56.126029 8192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:56.126068 8192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:56.126089 8192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:56.126102 8192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:56.132038 8192 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:43781
I20260925 07:20:56.132081 8320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:43781 every 8 connection(s)
I20260925 07:20:56.132404 8192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:56.138935 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8192
I20260925 07:20:56.139034 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:56.139292 4284 external_mini_cluster.cc:1428] Setting key 6906a086522a21bbe31ce9f822bdd701
I20260925 07:20:56.141734 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:56.133078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:52003 (local address 127.4.47.62:33307)
0925 07:20:56.133149 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:56.133151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:56.133362 (+ 211us) server_negotiation.cc:407] Connection header received
0925 07:20:56.134272 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:56.134273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:56.134283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:56.134340 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:56.134966 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.135555 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.136551 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.136760 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.136837 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:56.138930 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:56.138931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:56.138932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:56.138938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:56.140566 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.140930 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.140930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.140931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.140978 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.141236 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.141236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.141237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.141335 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:56.141385 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:56.141613 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 07:20:56.141671 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:20:56.141917 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:56.133075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33307 (local address 127.4.47.3:52003)
0925 07:20:56.133232 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:56.133267 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:56.134176 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:56.134417 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:56.134420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:56.134668 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:56.134871 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:56.134875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.135635 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.135636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:56.136460 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:56.136462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.136851 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.136851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:56.136946 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:56.137421 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:56.137428 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:56.138837 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:56.140632 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:56.140638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:56.140640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:56.140867 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:56.141034 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:56.141034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:56.141034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:56.141129 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:56.141391 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:56.141395 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:56.141507 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 07:20:56.141588 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:20:56.141713 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260925 07:20:56.142409 8321 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33307
I20260925 07:20:56.142501 8321 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:56.142637 8321 heartbeater.cc:511] Master 127.4.47.62:33307 requested a full tablet report, sending...
I20260925 07:20:56.143019 7862 ts_manager.cc:194] Registered new tserver with Master: dbfb927d085d4841b9546e9ddc45fb91 (127.4.47.3:43781)
I20260925 07:20:56.143411 7862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:52003
I20260925 07:20:56.151306 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:20:56.174691 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:56.166889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.4.47.1:36365)
0925 07:20:56.167128 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:56.167134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:56.167146 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:20:56.167182 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:56.167184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:56.167198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:56.167264 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:56.167801 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.168518 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.169220 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.169566 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.169619 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:56.171816 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:56.171828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:56.171829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:56.171841 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:56.173484 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.173850 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.173854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.173855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.173924 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.174125 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.174126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.174127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.174268 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:56.174436 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:56.174485 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 07:20:56.174569 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:56.182575 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:56.176079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.4.47.62:33307)
0925 07:20:56.176144 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:56.176146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:56.176157 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:20:56.176235 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:56.176236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:56.176247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:56.176305 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:56.176766 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.177260 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.177960 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:56.178178 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:56.178260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:56.180317 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:56.180318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:56.180319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:56.180326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:56.181450 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.181772 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.181773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.181774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.181838 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:56.182039 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:56.182040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:56.182040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:56.182183 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:56.182295 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:56.182391 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 07:20:56.182472 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 07:20:56.182787 7862 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35372
I20260925 07:20:56.183135 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7924
I20260925 07:20:56.188108 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8058
I20260925 07:20:56.192308 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8192
I20260925 07:20:56.196587 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 7830
2026-09-25T07:20:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1837 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[8333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[8333](info): set up 2 sockets
Sep 25 07:20:56 dist-test-slave-zw07 krb5kdc[8333](info): commencing operation
krb5kdc: starting...
W20260925 07:20:57.231340 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:20:57.350277 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45239
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:57.429996 8346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:57.430200 8346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:57.430222 8346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:57.431321 8346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:57.431355 8346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:57.431370 8346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:57.431382 8346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:57.431394 8346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:57.432811 8346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:57.433008 8346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:57.433175 8346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:57.437951 8354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.437956 8351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:57.438076 8346 server_base.cc:1034] running on GCE node
W20260925 07:20:57.437969 8352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:57.438409 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:57.438704 8346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:57.439845 8346 hybrid_clock.cc:648] HybridClock initialized: now 1790320857439815 us; error 47 us; skew 500 ppm
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:57.442325 8346 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:20:57.442687 8346 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:57.442735 8346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:57.442826 8346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:57.443905 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "463c9c91710745ee9b70777c51f23a4f"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "d00a9ad93b42587cde5773045535281d"
server_key_iv: "f39d6c9e445981a75ba747b74effc58f"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.444205 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "463c9c91710745ee9b70777c51f23a4f"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "d00a9ad93b42587cde5773045535281d"
server_key_iv: "f39d6c9e445981a75ba747b74effc58f"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.445406 8346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:20:57.446053 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.446241 8346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:57.446331 8346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "463c9c91710745ee9b70777c51f23a4f"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "d00a9ad93b42587cde5773045535281d"
server_key_iv: "f39d6c9e445981a75ba747b74effc58f"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.446413 8346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:57.457119 8346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:57.459556 8346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:57.459728 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:57.463735 8346 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:45239
I20260925 07:20:57.463776 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:45239 every 8 connection(s)
I20260925 07:20:57.464097 8346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:57.464890 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8346
I20260925 07:20:57.464978 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:57.465135 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:57.465183 4284 external_mini_cluster.cc:1428] Setting key fa20b0f311687256f47d592e7f1f0237
I20260925 07:20:57.468180 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: Bootstrap starting.
I20260925 07:20:57.468819 8413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:57.469099 8413 log.cc:824] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:57.469767 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: No bootstrap required, opened a new log
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:57.472038 8413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } }
I20260925 07:20:57.472222 8413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:57.472276 8413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463c9c91710745ee9b70777c51f23a4f, State: Initialized, Role: FOLLOWER
I20260925 07:20:57.472421 8413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } }
I20260925 07:20:57.472517 8413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:57.472551 8413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:57.472600 8413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:57.473335 8413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } }
I20260925 07:20:57.473455 8413 leader_election.cc:302] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 463c9c91710745ee9b70777c51f23a4f; no voters:
I20260925 07:20:57.473632 8413 leader_election.cc:288] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:57.473696 8418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:57.473888 8418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [term 1 LEADER]: Becoming Leader. State: Replica: 463c9c91710745ee9b70777c51f23a4f, State: Running, Role: LEADER
I20260925 07:20:57.473928 8413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:57.474048 8418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } }
I20260925 07:20:57.474359 8418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 463c9c91710745ee9b70777c51f23a4f. Latest consensus state: current_term: 1 leader_uuid: "463c9c91710745ee9b70777c51f23a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } } }
I20260925 07:20:57.474375 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "463c9c91710745ee9b70777c51f23a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463c9c91710745ee9b70777c51f23a4f" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45239 } } }
I20260925 07:20:57.474475 8418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:57.474699 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:57.475325 8425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:57.475971 8425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:57.476454 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:57.465726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.4.47.62:45239)
0925 07:20:57.465923 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:57.465931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:57.465956 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:20:57.466490 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:57.466497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:57.466546 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:57.466698 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:57.467491 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.468441 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.469324 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.469737 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.469858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:57.472569 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:57.472573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:57.472573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:57.472587 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:57.474668 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.475138 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.475140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.475141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.475254 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.475556 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.475558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.475558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.475833 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:57.475990 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:57.476167 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 07:20:57.476304 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260925 07:20:57.477469 8425 catalog_manager.cc:1347] Generated new cluster ID: f5b90633829e481381011ba3e308b363
I20260925 07:20:57.477523 8425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:57.491945 8425 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:57.492822 8425 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:57.495493 8425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: Generated new TSK 0
I20260925 07:20:57.495669 8425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:20:57.526448 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:57.643589 8439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:57.643903 8439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:57.643934 8439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:57.645140 8439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:57.645179 8439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:57.645207 8439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:57.646690 8439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:57.646900 8439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:57.647104 8439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:57.651997 8447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.652073 8445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.651997 8444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:57.652158 8439 server_base.cc:1034] running on GCE node
I20260925 07:20:57.652458 8439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:57.652704 8439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:57.653857 8439 hybrid_clock.cc:648] HybridClock initialized: now 1790320857653840 us; error 44 us; skew 500 ppm
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:57.656497 8439 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:20:57.656790 8439 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:57.656852 8439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:57.656948 8439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:57.657974 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "e6a334a9c22646f3a9215daf8e952cc9"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "12fabba48220fe223cfdb3e5c4a55f44"
server_key_iv: "b8ad24bb2b5232d8b72a64b97f3f0e2b"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.658269 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "e6a334a9c22646f3a9215daf8e952cc9"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "12fabba48220fe223cfdb3e5c4a55f44"
server_key_iv: "b8ad24bb2b5232d8b72a64b97f3f0e2b"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.659541 8439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:57.660181 8453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.660411 8439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.660547 8439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "e6a334a9c22646f3a9215daf8e952cc9"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "12fabba48220fe223cfdb3e5c4a55f44"
server_key_iv: "b8ad24bb2b5232d8b72a64b97f3f0e2b"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.660625 8439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:57.687726 8439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:57.689930 8439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:57.690083 8439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:57.690358 8439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:57.690692 8439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:57.690726 8439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.690747 8439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:57.690796 8439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.696609 8439 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:45399
I20260925 07:20:57.696722 8566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:45399 every 8 connection(s)
I20260925 07:20:57.697052 8439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:57.702324 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8439
I20260925 07:20:57.702416 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:57.702605 4284 external_mini_cluster.cc:1428] Setting key 38d0918ea80ad40816d799cfee8f756e
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:57.707134 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:57.697802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52243 (local address 127.4.47.62:45239)
0925 07:20:57.697876 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:57.697879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:57.698134 (+ 255us) server_negotiation.cc:407] Connection header received
0925 07:20:57.698677 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:57.698679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:57.698692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:57.698745 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:57.699779 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.700294 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.701311 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.701528 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.701600 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:57.704264 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:57.704265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:57.704265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:57.704272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:57.705684 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.706100 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.706102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.706103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.706162 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.706428 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.706430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.706430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.706570 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:57.706634 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:57.706929 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 07:20:57.707020 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:20:57.707324 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:57.697789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45239 (local address 127.4.47.1:52243)
0925 07:20:57.697973 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:57.698033 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:57.698554 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:57.698820 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:57.698824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:57.699155 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:57.699662 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:57.699667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.700393 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.700394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:57.701216 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:57.701218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.701619 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.701620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:57.701736 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:57.702262 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:57.702271 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:57.704161 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:57.705783 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:57.705790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:57.705792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:57.706021 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:57.706247 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:57.706248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:57.706248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:57.706358 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:57.706624 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:57.706631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:57.706777 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 07:20:57.706905 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 07:20:57.707087 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260925 07:20:57.707960 8567 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45239
I20260925 07:20:57.708070 8567 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:57.708220 8567 heartbeater.cc:511] Master 127.4.47.62:45239 requested a full tablet report, sending...
I20260925 07:20:57.708745 8377 ts_manager.cc:194] Registered new tserver with Master: e6a334a9c22646f3a9215daf8e952cc9 (127.4.47.1:45399)
I20260925 07:20:57.709371 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:52243
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:20:57.726449 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:57.807161 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:57.807405 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:57.807446 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:57.808602 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:57.808669 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:57.808715 8572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:57.810246 8572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:57.810524 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:57.810760 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:57.815827 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.815829 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.815826 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:57.815997 8572 server_base.cc:1034] running on GCE node
I20260925 07:20:57.816259 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:57.816524 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:57.817688 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790320857817674 us; error 47 us; skew 500 ppm
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:57.820256 8572 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:20:57.820605 8572 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:57.820677 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:57.820838 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:57.821920 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "094d353b71c64890b25d82c8fcaadbcb"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "da72069ec5dea87f47424152cfd5dc32"
server_key_iv: "cfe2b8184209602bde5ac0d55c16bf3c"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.822289 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "094d353b71c64890b25d82c8fcaadbcb"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "da72069ec5dea87f47424152cfd5dc32"
server_key_iv: "cfe2b8184209602bde5ac0d55c16bf3c"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.823719 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:57.824594 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.824780 8572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:57.824882 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "094d353b71c64890b25d82c8fcaadbcb"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "da72069ec5dea87f47424152cfd5dc32"
server_key_iv: "cfe2b8184209602bde5ac0d55c16bf3c"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.824965 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:57.849638 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:57.852095 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:57.852273 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:57.852512 8572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:57.852842 8572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:57.852896 8572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.852937 8572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:57.852964 8572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.858628 8572 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:36623
I20260925 07:20:57.858664 8699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:36623 every 8 connection(s)
I20260925 07:20:57.858960 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:57.861255 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8572
I20260925 07:20:57.861409 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:57.861610 4284 external_mini_cluster.cc:1428] Setting key f0582cb4eff482556d686b78e5fff618
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:57.869871 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:57.859742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:53877 (local address 127.4.47.62:45239)
0925 07:20:57.859822 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:57.859824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:57.860008 (+ 184us) server_negotiation.cc:407] Connection header received
0925 07:20:57.860463 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:57.860463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:57.860474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:57.860524 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:57.861646 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.862434 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.864044 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.864289 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.864382 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:57.866777 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:57.866778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:57.866779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:57.866787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:57.868483 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.868873 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.868874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.868875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.868940 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:57.869192 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:57.869193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:57.869194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:57.869336 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:57.869415 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:57.869665 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 07:20:57.869747 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:20:57.869901 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:57.859702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45239 (local address 127.4.47.2:53877)
0925 07:20:57.859889 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:57.859927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:57.860384 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:57.860588 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:57.860591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:57.860845 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:57.861550 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:57.861557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.862522 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.862523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:57.863950 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:57.863958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:57.864391 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:57.864392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:57.864534 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:57.865067 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:57.865077 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:57.866683 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:57.868564 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:57.868569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:57.868571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:57.868779 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:57.869022 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:57.869022 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:57.869023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:57.869124 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:57.869417 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:57.869420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:57.869533 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 07:20:57.869612 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:20:57.869725 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
I20260925 07:20:57.870653 8700 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45239
I20260925 07:20:57.870764 8700 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:57.870918 8700 heartbeater.cc:511] Master 127.4.47.62:45239 requested a full tablet report, sending...
I20260925 07:20:57.871367 8377 ts_manager.cc:194] Registered new tserver with Master: 094d353b71c64890b25d82c8fcaadbcb (127.4.47.2:36623)
I20260925 07:20:57.871728 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:53877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:20:57.889384 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:20:57.970858 8705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:57.971055 8705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:57.971150 8705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:57.972289 8705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:57.972326 8705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:57.972378 8705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:57.973904 8705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:57.974139 8705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:57.974368 8705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:57.979228 8713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.979265 8711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:57.979300 8710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:57.979404 8705 server_base.cc:1034] running on GCE node
I20260925 07:20:57.979641 8705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:57.979893 8705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:57.981053 8705 hybrid_clock.cc:648] HybridClock initialized: now 1790320857981020 us; error 46 us; skew 500 ppm
Sep 25 07:20:57 dist-test-slave-zw07 krb5kdc[8333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:20:57.983584 8705 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:20:57.983953 8705 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:57.984023 8705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:57.984177 8705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:57.985190 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "f355ff7a6c824db4891dc6f4827f8518"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "b2a23086f861523827aa937fd9ce85d2"
server_key_iv: "29124d4739cb578a8f08568e2fdc4e12"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.985543 8705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "f355ff7a6c824db4891dc6f4827f8518"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "b2a23086f861523827aa937fd9ce85d2"
server_key_iv: "29124d4739cb578a8f08568e2fdc4e12"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.986838 8705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:20:57.987633 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.987861 8705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:20:57.987969 8705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "f355ff7a6c824db4891dc6f4827f8518"
format_stamp: "Formatted at 2026-09-25 07:20:57 on dist-test-slave-zw07"
server_key: "b2a23086f861523827aa937fd9ce85d2"
server_key_iv: "29124d4739cb578a8f08568e2fdc4e12"
server_key_version: "encryptionkey@0"
I20260925 07:20:57.988044 8705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:57.996003 8705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:57.997887 8705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:57.998029 8705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:57.998250 8705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:57.998556 8705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:57.998589 8705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:57.998610 8705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:57.998656 8705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.004360 8705 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:39501
I20260925 07:20:58.004429 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:39501 every 8 connection(s)
I20260925 07:20:58.004681 8705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:20:58 dist-test-slave-zw07 krb5kdc[8333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320857, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:20:58.013998 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.005412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:34161 (local address 127.4.47.62:45239)
0925 07:20:58.005460 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.005462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.005631 (+ 169us) server_negotiation.cc:407] Connection header received
0925 07:20:58.006123 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.006124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.006133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.006196 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.007191 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.007732 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.008726 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.008919 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.008995 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.011426 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.011428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.011428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:58.011436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.012767 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:58.013155 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:58.013155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:58.013156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:58.013203 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:58.013468 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:58.013469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:58.013469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:58.013570 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.013620 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.013867 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.013929 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:20:58.014271 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.005364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45239 (local address 127.4.47.3:34161)
0925 07:20:58.005520 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.005561 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.006038 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.006268 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.006272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.006538 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.007053 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.007059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.007833 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.007833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.008643 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.008646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.009044 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.009045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.009180 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.009822 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:20:58.009832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.011330 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.012863 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:58.012869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:58.012871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:58.013087 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:58.013276 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:20:58.013276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:20:58.013277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:20:58.013411 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:20:58.013626 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.013630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.013750 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.013838 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.013986 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 07:20:58.014297 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8705
I20260925 07:20:58.014412 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:58.014598 4284 external_mini_cluster.cc:1428] Setting key 98881aacd24b78120d80b955f3e4aff8
I20260925 07:20:58.014912 8833 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45239
I20260925 07:20:58.015038 8833 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:58.015250 8833 heartbeater.cc:511] Master 127.4.47.62:45239 requested a full tablet report, sending...
I20260925 07:20:58.015645 8377 ts_manager.cc:194] Registered new tserver with Master: f355ff7a6c824db4891dc6f4827f8518 (127.4.47.3:39501)
I20260925 07:20:58.016002 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:34161
I20260925 07:20:58.026393 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:20:58.033594 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.027844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.4.47.62:45239)
0925 07:20:58.027922 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.027924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.027991 (+ 67us) server_negotiation.cc:407] Connection header received
0925 07:20:58.028071 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.028072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.028084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.028130 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.028727 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.029209 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.029914 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.030087 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.030137 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.031396 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.031397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.031397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:20:58.031402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.032581 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:58.032915 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:58.032916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:58.032916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:58.032965 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:20:58.033123 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:20:58.033124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:20:58.033124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:20:58.033271 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.033345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.033439 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.033497 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 07:20:58.035248 8377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:58.036844 8377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:58.043663 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.4.47.3:39501)
0925 07:20:58.039615 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.039621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.039631 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:58.039675 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.039677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.039694 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.039796 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:58.039973 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041121 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.042531 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043221 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043296 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043365 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.043512 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.043555 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
I20260925 07:20:58.043653 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:39501 (local address 127.0.0.1:60546)
0925 07:20:58.039363 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.039408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.039470 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.039781 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.039784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.039786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:58.039859 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.039861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041249 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.041250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.042442 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.042444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043307 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.043385 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043400 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.043488 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.043551 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":75,"thread_start_us":69,"threads_started":1}
I20260925 07:20:58.043843 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45399 (local address 127.0.0.1:44126)
0925 07:20:58.039564 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.039576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.039632 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.040056 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.040057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.040059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:58.040107 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.040109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041382 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.041383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.042711 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.042713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043540 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.043590 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043593 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.043622 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.043653 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":75,"thread_start_us":95,"threads_started":1}
I20260925 07:20:58.043984 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:36623 (local address 127.0.0.1:48986)
0925 07:20:58.039618 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.039627 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.039669 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.040318 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.040319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.040321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:58.040354 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.040355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041232 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.041233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.042711 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.042713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043737 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.043802 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043819 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.043870 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.043915 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":133,"thread_start_us":111,"threads_started":1}
I20260925 07:20:58.044138 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.4.47.1:45399)
0925 07:20:58.039861 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.039865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.039875 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:58.039929 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.039931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.039946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.040177 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:58.040254 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041273 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.042814 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043450 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043692 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043986 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.044014 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.044057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260925 07:20:58.044199 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.039892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.4.47.2:36623)
0925 07:20:58.040089 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.040093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.040103 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:58.040139 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.040141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.040157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.040230 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:58.040431 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.041105 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.042800 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.043581 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.043646 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.043699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.044086 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.044126 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260925 07:20:58.044526 8767 tablet_service.cc:1467] Processing CreateTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 (DEFAULT_TABLE table=test-table [id=80c8e75b5e3048b0897f6343c288079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.044864 8767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57edcf9427c4f8abdd0820f97bb3b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.044945 8634 tablet_service.cc:1467] Processing CreateTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 (DEFAULT_TABLE table=test-table [id=80c8e75b5e3048b0897f6343c288079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.045222 8634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57edcf9427c4f8abdd0820f97bb3b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.047401 8850 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Bootstrap starting.
I20260925 07:20:58.047709 8851 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Bootstrap starting.
I20260925 07:20:58.047721 8501 tablet_service.cc:1467] Processing CreateTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 (DEFAULT_TABLE table=test-table [id=80c8e75b5e3048b0897f6343c288079a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.047982 8850 tablet_bootstrap.cc:654] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.048034 8501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57edcf9427c4f8abdd0820f97bb3b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.048363 8851 tablet_bootstrap.cc:654] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.048416 8850 log.cc:824] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.049122 8851 log.cc:824] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.049160 8850 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: No bootstrap required, opened a new log
I20260925 07:20:58.049224 8850 ts_tablet_manager.cc:1397] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:58.049903 8851 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: No bootstrap required, opened a new log
I20260925 07:20:58.049962 8851 ts_tablet_manager.cc:1397] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:58.051021 8850 raft_consensus.cc:348] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.051250 8850 raft_consensus.cc:374] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.051296 8850 raft_consensus.cc:729] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f355ff7a6c824db4891dc6f4827f8518, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.051411 8850 consensus_queue.cc:260] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.051671 8850 ts_tablet_manager.cc:1428] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:20:58.051661 8854 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: Bootstrap starting.
I20260925 07:20:58.051868 8833 heartbeater.cc:503] Master 127.4.47.62:45239 was elected leader, sending a full tablet report...
I20260925 07:20:58.052109 8851 raft_consensus.cc:348] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.052282 8851 raft_consensus.cc:374] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.052300 8854 tablet_bootstrap.cc:654] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.052333 8851 raft_consensus.cc:729] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094d353b71c64890b25d82c8fcaadbcb, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.052484 8851 consensus_queue.cc:260] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.052814 8851 ts_tablet_manager.cc:1428] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:20:58.052824 8700 heartbeater.cc:503] Master 127.4.47.62:45239 was elected leader, sending a full tablet report...
I20260925 07:20:58.052896 8854 log.cc:824] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.053712 8854 tablet_bootstrap.cc:492] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: No bootstrap required, opened a new log
I20260925 07:20:58.053778 8854 ts_tablet_manager.cc:1397] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:58.055235 8854 raft_consensus.cc:348] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.055366 8854 raft_consensus.cc:374] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.055388 8854 raft_consensus.cc:729] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a334a9c22646f3a9215daf8e952cc9, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.055465 8854 consensus_queue.cc:260] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.055660 8854 ts_tablet_manager.cc:1428] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:20:58.055727 8567 heartbeater.cc:503] Master 127.4.47.62:45239 was elected leader, sending a full tablet report...
W20260925 07:20:58.109525 8701 tablet.cc:2369] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:58.175276 8855 raft_consensus.cc:482] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:58.175417 8855 raft_consensus.cc:504] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.175772 8855 leader_election.cc:288] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 094d353b71c64890b25d82c8fcaadbcb (127.4.47.2:36623), e6a334a9c22646f3a9215daf8e952cc9 (127.4.47.1:45399)
I20260925 07:20:58.178763 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.176020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:38101 (local address 127.4.47.1:45399)
0925 07:20:58.176116 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.176118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.176254 (+ 136us) server_negotiation.cc:407] Connection header received
0925 07:20:58.176406 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.176407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.176420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.176478 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:58.176652 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.177090 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178052 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.178412 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178467 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.178497 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.178649 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.178694 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 07:20:58.178763 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.175976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45399 (local address 127.4.47.3:38101)
0925 07:20:58.176166 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.176176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.176239 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.176490 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.176491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.176492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:58.176579 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.176581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.177202 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.177203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.177957 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.177959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.178497 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.178569 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.178572 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.178622 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.178651 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260925 07:20:58.178814 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.176044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:39303 (local address 127.4.47.2:36623)
0925 07:20:58.176094 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.176097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.176111 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:20:58.176207 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.176209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.176223 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.176284 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:20:58.176563 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.177066 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178056 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.178422 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178504 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.178514 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.178703 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.178742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:58.178872 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.175965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:36623 (local address 127.4.47.3:39303)
0925 07:20:58.176061 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.176073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.176124 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.176404 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.176405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.176407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:20:58.176485 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.176487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.177150 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.177151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.177957 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:58.177959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.178525 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.178526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:58.178624 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.178627 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.178685 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.178721 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":19}
I20260925 07:20:58.179203 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57edcf9427c4f8abdd0820f97bb3b67" candidate_uuid: "f355ff7a6c824db4891dc6f4827f8518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "094d353b71c64890b25d82c8fcaadbcb" is_pre_election: true
I20260925 07:20:58.179174 8521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57edcf9427c4f8abdd0820f97bb3b67" candidate_uuid: "f355ff7a6c824db4891dc6f4827f8518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a334a9c22646f3a9215daf8e952cc9" is_pre_election: true
I20260925 07:20:58.179344 8521 raft_consensus.cc:2454] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f355ff7a6c824db4891dc6f4827f8518 in term 0.
I20260925 07:20:58.179344 8654 raft_consensus.cc:2454] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f355ff7a6c824db4891dc6f4827f8518 in term 0.
I20260925 07:20:58.179570 8720 leader_election.cc:302] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094d353b71c64890b25d82c8fcaadbcb, f355ff7a6c824db4891dc6f4827f8518; no voters:
I20260925 07:20:58.179744 8855 raft_consensus.cc:2781] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:58.179801 8855 raft_consensus.cc:482] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:58.179824 8855 raft_consensus.cc:3037] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:58.180565 8855 raft_consensus.cc:504] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.180714 8855 leader_election.cc:288] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [CANDIDATE]: Term 1 election: Requested vote from peers 094d353b71c64890b25d82c8fcaadbcb (127.4.47.2:36623), e6a334a9c22646f3a9215daf8e952cc9 (127.4.47.1:45399)
I20260925 07:20:58.180889 8521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57edcf9427c4f8abdd0820f97bb3b67" candidate_uuid: "f355ff7a6c824db4891dc6f4827f8518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a334a9c22646f3a9215daf8e952cc9"
I20260925 07:20:58.180891 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57edcf9427c4f8abdd0820f97bb3b67" candidate_uuid: "f355ff7a6c824db4891dc6f4827f8518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "094d353b71c64890b25d82c8fcaadbcb"
I20260925 07:20:58.180972 8654 raft_consensus.cc:3037] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:58.180972 8521 raft_consensus.cc:3037] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:58.181864 8521 raft_consensus.cc:2454] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f355ff7a6c824db4891dc6f4827f8518 in term 1.
I20260925 07:20:58.181874 8654 raft_consensus.cc:2454] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f355ff7a6c824db4891dc6f4827f8518 in term 1.
I20260925 07:20:58.182056 8723 leader_election.cc:302] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6a334a9c22646f3a9215daf8e952cc9, f355ff7a6c824db4891dc6f4827f8518; no voters:
I20260925 07:20:58.182147 8855 raft_consensus.cc:2781] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:58.182327 8855 raft_consensus.cc:686] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 1 LEADER]: Becoming Leader. State: Replica: f355ff7a6c824db4891dc6f4827f8518, State: Running, Role: LEADER
I20260925 07:20:58.182440 8855 consensus_queue.cc:237] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } }
I20260925 07:20:58.183447 8377 catalog_manager.cc:5495] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 reported cstate change: term changed from 0 to 1, leader changed from <none> to f355ff7a6c824db4891dc6f4827f8518 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "f355ff7a6c824db4891dc6f4827f8518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f355ff7a6c824db4891dc6f4827f8518" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 } health_report { overall_health: UNKNOWN } } }
I20260925 07:20:58.190675 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.188499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.4.47.3:39501)
0925 07:20:58.188540 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.188542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.188553 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:20:58.188629 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.188629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.188645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.188702 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:20:58.188804 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.189244 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.189938 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:58.190112 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:58.190162 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:58.190345 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:20:58.190436 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:20:58.190508 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.190553 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.190571 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:20:58.192435 8654 raft_consensus.cc:1261] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 1 FOLLOWER]: Refusing update from remote peer f355ff7a6c824db4891dc6f4827f8518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:58.192435 8521 raft_consensus.cc:1261] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 1 FOLLOWER]: Refusing update from remote peer f355ff7a6c824db4891dc6f4827f8518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:58.192811 8860 consensus_queue.cc:1035] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094d353b71c64890b25d82c8fcaadbcb" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 36623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:58.192925 8855 consensus_queue.cc:1035] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a334a9c22646f3a9215daf8e952cc9" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:58.195022 8863 mvcc.cc:205] Tried to move back new op lower bound from 7333154235153584128 to 7333154235116175360. Current Snapshot: MvccSnapshot[applied={T|T < 7333154235153584128}]
I20260925 07:20:58.197295 8377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:58.197400 8377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:58.198149 8377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 463c9c91710745ee9b70777c51f23a4f: Sending DeleteTablet for 3 replicas of tablet c57edcf9427c4f8abdd0820f97bb3b67
I20260925 07:20:58.198462 8767 tablet_service.cc:1514] Processing DeleteTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:58 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60546
I20260925 07:20:58.198495 8501 tablet_service.cc:1514] Processing DeleteTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:58 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44126
I20260925 07:20:58.198529 8634 tablet_service.cc:1514] Processing DeleteTablet for tablet c57edcf9427c4f8abdd0820f97bb3b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:58 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:48986
I20260925 07:20:58.198678 8874 tablet_replica.cc:330] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9: stopping tablet replica
I20260925 07:20:58.198773 8874 raft_consensus.cc:2229] T c57edcf9427c4f8abdd0820f97bb3b67 P e6a334a9c22646f3a9215daf8e952cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:58.198944 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8439
I20260925 07:20:58.200014 8875 tablet_replica.cc:330] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: stopping tablet replica
I20260925 07:20:58.200110 8875 raft_consensus.cc:2229] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:58.200215 8875 raft_consensus.cc:2258] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:58.200642 8875 ts_tablet_manager.cc:1905] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:58.202874 8875 ts_tablet_manager.cc:1918] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:58.202963 8875 log.cc:1196] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/c57edcf9427c4f8abdd0820f97bb3b67
I20260925 07:20:58.203184 8873 tablet_replica.cc:330] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: stopping tablet replica
I20260925 07:20:58.203274 8875 ts_tablet_manager.cc:1939] T c57edcf9427c4f8abdd0820f97bb3b67 P 094d353b71c64890b25d82c8fcaadbcb: Deleting consensus metadata
I20260925 07:20:58.203406 8873 raft_consensus.cc:2229] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:58.203555 8873 raft_consensus.cc:2258] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:58.203722 8361 catalog_manager.cc:4841] TS 094d353b71c64890b25d82c8fcaadbcb (127.4.47.2:36623): tablet c57edcf9427c4f8abdd0820f97bb3b67 (table test-table [id=80c8e75b5e3048b0897f6343c288079a]) successfully deleted
I20260925 07:20:58.204097 8873 ts_tablet_manager.cc:1905] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:20:58.205003 8364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:45399 (error 108)
I20260925 07:20:58.205104 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8572
I20260925 07:20:58.206130 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.206003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45399 (local address 127.0.0.1:44136)
0925 07:20:58.206044 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.206093 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:45399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 07:20:58.206125 8873 ts_tablet_manager.cc:1918] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:58.206200 8873 log.cc:1196] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/c57edcf9427c4f8abdd0820f97bb3b67
W20260925 07:20:58.206271 8364 catalog_manager.cc:4584] TS e6a334a9c22646f3a9215daf8e952cc9 (127.4.47.1:45399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c57edcf9427c4f8abdd0820f97bb3b67: Network error: Client connection negotiation failed: client connection to 127.4.47.1:45399: connect: Connection refused (error 111)
I20260925 07:20:58.206486 8873 ts_tablet_manager.cc:1939] T c57edcf9427c4f8abdd0820f97bb3b67 P f355ff7a6c824db4891dc6f4827f8518: Deleting consensus metadata
I20260925 07:20:58.206921 8361 catalog_manager.cc:4841] TS f355ff7a6c824db4891dc6f4827f8518 (127.4.47.3:39501): tablet c57edcf9427c4f8abdd0820f97bb3b67 (table test-table [id=80c8e75b5e3048b0897f6343c288079a]) successfully deleted
I20260925 07:20:58.211172 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8705
I20260925 07:20:58.216413 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8346
2026-09-25T07:20:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2020 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T07:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:58Z Disabled control of system clock
I20260925 07:20:58.243691 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45635
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:20:58.324934 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:58.325143 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:58.325165 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:58.326385 8880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:58.326432 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:58.326445 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:58.326485 8880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:58.326514 8880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:20:58.328039 8880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:58.328310 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:58.328617 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:58.333489 8885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.333638 8886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.333774 8880 server_base.cc:1034] running on GCE node
W20260925 07:20:58.333642 8888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.334110 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:58.334363 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:58.335533 8880 hybrid_clock.cc:648] HybridClock initialized: now 1790320858335509 us; error 35 us; skew 500 ppm
I20260925 07:20:58.336758 8880 webserver.cc:460] Webserver started at http://127.4.47.62:43603/ using document root <none> and password file <none>
I20260925 07:20:58.337002 8880 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:58.337070 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:58.337215 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:58.338317 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "c4ab0c7144874dc1b7b65a85e9438978"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7c6b340aedd6227626c54ee0fe468263"
server_key_iv: "ed115ba59d54025d860e6b14772f05f3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.338680 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "c4ab0c7144874dc1b7b65a85e9438978"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7c6b340aedd6227626c54ee0fe468263"
server_key_iv: "ed115ba59d54025d860e6b14772f05f3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.340076 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:20:58.340785 8894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.340973 8880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:58.341078 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c4ab0c7144874dc1b7b65a85e9438978"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7c6b340aedd6227626c54ee0fe468263"
server_key_iv: "ed115ba59d54025d860e6b14772f05f3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.341152 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:58.345360 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:58.345657 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:58.345791 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:58.349951 8880 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:45635
I20260925 07:20:58.349983 8946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:45635 every 8 connection(s)
I20260925 07:20:58.350327 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:58.351025 8947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.353785 8947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: Bootstrap starting.
I20260925 07:20:58.354312 8947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.354549 8947 log.cc:824] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.355233 8947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: No bootstrap required, opened a new log
I20260925 07:20:58.356719 8947 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } }
I20260925 07:20:58.356868 8947 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.356892 8947 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ab0c7144874dc1b7b65a85e9438978, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.357030 8947 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } }
I20260925 07:20:58.357105 8947 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:58.357127 8947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:58.357155 8947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:58.357626 8947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } }
I20260925 07:20:58.357709 8947 leader_election.cc:302] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ab0c7144874dc1b7b65a85e9438978; no voters:
I20260925 07:20:58.357909 8947 leader_election.cc:288] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:58.357952 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8880
I20260925 07:20:58.357973 8950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:58.358062 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:58.358124 8950 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [term 1 LEADER]: Becoming Leader. State: Replica: c4ab0c7144874dc1b7b65a85e9438978, State: Running, Role: LEADER
I20260925 07:20:58.358176 8947 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:58.358250 4284 external_mini_cluster.cc:1428] Setting key 56411e20c7fc085c0cef64cad46ca849
I20260925 07:20:58.358224 8950 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } }
I20260925 07:20:58.358564 8950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ab0c7144874dc1b7b65a85e9438978. Latest consensus state: current_term: 1 leader_uuid: "c4ab0c7144874dc1b7b65a85e9438978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } } }
I20260925 07:20:58.358570 8951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ab0c7144874dc1b7b65a85e9438978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ab0c7144874dc1b7b65a85e9438978" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45635 } } }
I20260925 07:20:58.358655 8950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:58.358682 8951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:58.359325 8956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:58.359892 8956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:58.361367 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.358996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.4.47.62:45635)
0925 07:20:58.359758 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.359769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.359803 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:20:58.359923 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.359930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.359981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.360133 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.360863 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.360867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.360868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.360873 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.360922 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.360937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.361029 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.361122 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.361191 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":108,"threads_started":1}
I20260925 07:20:58.361583 8956 catalog_manager.cc:1347] Generated new cluster ID: 73684554aeb74bdcb24a6de4777f8a9b
I20260925 07:20:58.361635 8956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:58.379199 8956 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:58.379408 8956 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:58.387593 8956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: Generated new TSK 0
I20260925 07:20:58.387810 8956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:20:58.394809 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--builtin_ntp_servers=127.4.47.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:20:58.476596 8971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:58.476814 8971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:58.476836 8971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:58.478027 8971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:58.478071 8971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:58.478097 8971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:20:58.479591 8971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:58.479866 8971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:58.480197 8971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:58.485258 8976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.485373 8971 server_base.cc:1034] running on GCE node
W20260925 07:20:58.485271 8979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.485414 8977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.485702 8971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:58.485908 8971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:58.487123 8971 hybrid_clock.cc:648] HybridClock initialized: now 1790320858487112 us; error 40 us; skew 500 ppm
I20260925 07:20:58.488309 8971 webserver.cc:460] Webserver started at http://127.4.47.1:36427/ using document root <none> and password file <none>
I20260925 07:20:58.488574 8971 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:58.488646 8971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:58.488792 8971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:58.489907 8971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "4a19950ae53e4732b4e54f2b1b7e0438"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "1a355af4347a52c7e4d89b038fe986c0"
server_key_iv: "68068f34573fd2a454868308de1b06c6"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.490264 8971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "4a19950ae53e4732b4e54f2b1b7e0438"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "1a355af4347a52c7e4d89b038fe986c0"
server_key_iv: "68068f34573fd2a454868308de1b06c6"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.491658 8971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:58.492478 8985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.492668 8971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:58.492794 8971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "4a19950ae53e4732b4e54f2b1b7e0438"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "1a355af4347a52c7e4d89b038fe986c0"
server_key_iv: "68068f34573fd2a454868308de1b06c6"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.492887 8971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:58.496742 8971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:58.497033 8971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:58.497184 8971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:58.497406 8971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:58.497723 8971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:58.497781 8971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.497824 8971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:58.497855 8971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.504396 8971 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:39495
I20260925 07:20:58.504442 9098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:39495 every 8 connection(s)
I20260925 07:20:58.504796 8971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:20:58.507951 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.505560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:33669 (local address 127.4.47.62:45635)
0925 07:20:58.505627 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.505630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.505945 (+ 315us) server_negotiation.cc:407] Connection header received
0925 07:20:58.506394 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.506395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.506402 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.506462 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.507722 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.507723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.507724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.507727 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.507743 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.507747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.507806 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.507881 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.507897 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:20:58.508129 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.505537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45635 (local address 127.4.47.1:33669)
0925 07:20:58.505811 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.505845 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.506307 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.506513 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.506516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.506524 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.507576 (+ 1052us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.507577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.507605 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.507634 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.507811 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.507814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.507817 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.507877 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.507926 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260925 07:20:58.508639 9099 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45635
I20260925 07:20:58.508730 9099 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:58.508843 9099 heartbeater.cc:511] Master 127.4.47.62:45635 requested a full tablet report, sending...
I20260925 07:20:58.509330 8911 ts_manager.cc:194] Registered new tserver with Master: 4a19950ae53e4732b4e54f2b1b7e0438 (127.4.47.1:39495)
I20260925 07:20:58.509733 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 8971
I20260925 07:20:58.509858 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:20:58.510044 4284 external_mini_cluster.cc:1428] Setting key 301f70de1e5078edcef2b129a5c3acea
I20260925 07:20:58.510943 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--builtin_ntp_servers=127.4.47.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:20:58.591483 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:58.591712 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:58.591750 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:58.592935 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:58.593004 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:58.593050 9102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:20:58.594441 9102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:58.594724 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:58.595027 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:58.600242 9107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.600239 9110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.600389 9108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.600456 9102 server_base.cc:1034] running on GCE node
I20260925 07:20:58.600723 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:58.600968 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:58.602129 9102 hybrid_clock.cc:648] HybridClock initialized: now 1790320858602124 us; error 48 us; skew 500 ppm
I20260925 07:20:58.603434 9102 webserver.cc:460] Webserver started at http://127.4.47.2:44091/ using document root <none> and password file <none>
I20260925 07:20:58.603688 9102 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:58.603753 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:58.603904 9102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:58.604986 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "8c986e07fb4f48b89c26208f4c8966ab"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "bacddf0122ed74ab84b8201023556860"
server_key_iv: "e80da35040094db2b824c13ca7012ed3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.605346 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "8c986e07fb4f48b89c26208f4c8966ab"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "bacddf0122ed74ab84b8201023556860"
server_key_iv: "e80da35040094db2b824c13ca7012ed3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.606703 9102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:20:58.607489 9116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.607658 9102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.607757 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "8c986e07fb4f48b89c26208f4c8966ab"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "bacddf0122ed74ab84b8201023556860"
server_key_iv: "e80da35040094db2b824c13ca7012ed3"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.607833 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:58.611162 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:58.611425 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:58.611567 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:58.611768 9102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:58.612066 9102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:58.612118 9102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.612156 9102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:58.612185 9102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.617974 9102 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:38583
I20260925 07:20:58.618008 9229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:38583 every 8 connection(s)
I20260925 07:20:58.618335 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:20:58.621217 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.619124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56483 (local address 127.4.47.62:45635)
0925 07:20:58.619179 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.619181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.619397 (+ 216us) server_negotiation.cc:407] Connection header received
0925 07:20:58.619843 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.619844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.619849 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.619903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.621041 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.621042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.621042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.621045 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.621055 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.621058 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.621098 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.621144 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.621155 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:20:58.621426 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.619106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45635 (local address 127.4.47.2:56483)
0925 07:20:58.619283 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.619330 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.619758 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.619953 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.619956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.619964 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.620914 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.620914 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.620942 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.620970 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.621097 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.621100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.621102 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.621152 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.621210 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":100,"threads_started":1}
I20260925 07:20:58.621874 9230 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45635
I20260925 07:20:58.621959 9230 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:58.622063 9230 heartbeater.cc:511] Master 127.4.47.62:45635 requested a full tablet report, sending...
I20260925 07:20:58.622385 8911 ts_manager.cc:194] Registered new tserver with Master: 8c986e07fb4f48b89c26208f4c8966ab (127.4.47.2:38583)
I20260925 07:20:58.625403 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9102
I20260925 07:20:58.625484 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:20:58.625624 4284 external_mini_cluster.cc:1428] Setting key 90e7f52b08c75e81ae920a3a097f424a
I20260925 07:20:58.626421 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--builtin_ntp_servers=127.4.47.20:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:20:58.709578 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:58.709805 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:58.709843 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:58.711127 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:58.711184 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:58.711226 9233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:20:58.712603 9233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:58.712857 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:58.713140 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:58.718152 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.718204 9239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:58.718210 9238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:58.718366 9233 server_base.cc:1034] running on GCE node
I20260925 07:20:58.718544 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:58.718734 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:58.719874 9233 hybrid_clock.cc:648] HybridClock initialized: now 1790320858719855 us; error 34 us; skew 500 ppm
I20260925 07:20:58.721046 9233 webserver.cc:460] Webserver started at http://127.4.47.3:41903/ using document root <none> and password file <none>
I20260925 07:20:58.721297 9233 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:58.721343 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:58.721501 9233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:58.722529 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "97dcb977b18a407195d677189995a93d"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7007796e4e73c638d19de9b9da532fa1"
server_key_iv: "a54823c89af4d8463374e862b8419926"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.722883 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "97dcb977b18a407195d677189995a93d"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7007796e4e73c638d19de9b9da532fa1"
server_key_iv: "a54823c89af4d8463374e862b8419926"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.724220 9233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:58.724916 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.725087 9233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:58.725174 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "97dcb977b18a407195d677189995a93d"
format_stamp: "Formatted at 2026-09-25 07:20:58 on dist-test-slave-zw07"
server_key: "7007796e4e73c638d19de9b9da532fa1"
server_key_iv: "a54823c89af4d8463374e862b8419926"
server_key_version: "encryptionkey@0"
I20260925 07:20:58.725255 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:58.728587 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:58.728843 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:58.728986 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:58.729192 9233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:20:58.729487 9233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:20:58.729537 9233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.729575 9233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:20:58.729604 9233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:58.735500 9233 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:35279
I20260925 07:20:58.735544 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:35279 every 8 connection(s)
I20260925 07:20:58.735870 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:20:58.738767 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.736569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:60279 (local address 127.4.47.62:45635)
0925 07:20:58.736637 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.736641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.736881 (+ 240us) server_negotiation.cc:407] Connection header received
0925 07:20:58.737327 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.737328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.737333 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.737396 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.738598 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.738598 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.738599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.738601 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.738612 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.738615 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.738658 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.738704 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.738716 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:20:58.738997 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.736564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45635 (local address 127.4.47.3:60279)
0925 07:20:58.736748 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.736782 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.737250 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.737468 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.737471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.737478 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.738464 (+ 986us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.738464 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.738492 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.738520 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.738657 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.738659 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.738661 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.738712 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.738757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260925 07:20:58.739554 9361 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45635
I20260925 07:20:58.739655 9361 heartbeater.cc:464] Registering TS with master...
I20260925 07:20:58.739795 9361 heartbeater.cc:511] Master 127.4.47.62:45635 requested a full tablet report, sending...
I20260925 07:20:58.740162 8911 ts_manager.cc:194] Registered new tserver with Master: 97dcb977b18a407195d677189995a93d (127.4.47.3:35279)
I20260925 07:20:58.740715 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9233
I20260925 07:20:58.740828 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:20:58.741012 4284 external_mini_cluster.cc:1428] Setting key 5a2d53446459ec12fbb7c393f079058b
I20260925 07:20:58.741811 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:20:58.744063 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.743095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.4.47.62:45635)
0925 07:20:58.743149 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.743151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.743220 (+ 69us) server_negotiation.cc:407] Connection header received
0925 07:20:58.743274 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.743274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.743278 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.743325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.743919 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.743919 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.743920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.743922 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.743933 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.743936 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.743977 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.744009 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.744020 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:20:58.745604 8911 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:20:58.747123 8911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:20:58.750658 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38583 (local address 127.0.0.1:43222)
0925 07:20:58.749483 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.749491 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.749528 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.749686 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.749691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.749697 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.750174 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.750174 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.750187 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.750206 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.750487 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.750492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.750495 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.750556 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.750571 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260925 07:20:58.750877 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.4.47.1:39495)
0925 07:20:58.749562 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.749567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.749578 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:20:58.749606 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.749608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.749618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.749699 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.750466 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.750473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.750473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.750478 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.750508 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.750529 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.750581 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.750691 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.750706 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260925 07:20:58.750962 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.4.47.2:38583)
0925 07:20:58.749469 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.749474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.749527 (+ 53us) server_negotiation.cc:407] Connection header received
0925 07:20:58.749586 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.749588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.749598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.749672 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.750330 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.750338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.750339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.750343 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.750382 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.750406 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.750461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.750838 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.750866 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260925 07:20:58.751037 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:39495 (local address 127.0.0.1:43442)
0925 07:20:58.749304 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.749343 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.749399 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.750037 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.750037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.750040 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.750365 (+ 325us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.750366 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.750372 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.750382 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.750604 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.750604 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.750605 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.750644 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.750654 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260925 07:20:58.751507 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:35279 (local address 127.0.0.1:38528)
0925 07:20:58.749621 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:58.749631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:58.749676 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:58.750592 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:58.750593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:58.750595 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:58.751154 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:58.751154 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:58.751161 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:58.751174 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:58.751394 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:58.751395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:58.751395 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 07:20:58.751436 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 07:20:58.751448 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":81,"threads_started":1}
I20260925 07:20:58.751693 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:58.749466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.4.47.3:35279)
0925 07:20:58.750218 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:58.750222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:58.750232 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:58.750257 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:58.750259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:58.750266 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:58.750340 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:58.751253 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:58.751259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:58.751259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:58.751263 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:58.751296 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:58.751315 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:58.751370 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:58.751600 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 07:20:58.751615 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":75,"threads_started":1}
I20260925 07:20:58.751592 9164 tablet_service.cc:1467] Processing CreateTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 (DEFAULT_TABLE table=test-table [id=42cb57346f9b403bad80ba93c70e0daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.751713 9033 tablet_service.cc:1467] Processing CreateTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 (DEFAULT_TABLE table=test-table [id=42cb57346f9b403bad80ba93c70e0daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.751968 9164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4816e0f9634843e491f6f99ef6b77e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.752068 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4816e0f9634843e491f6f99ef6b77e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.754462 9375 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Bootstrap starting.
I20260925 07:20:58.754468 9376 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Bootstrap starting.
I20260925 07:20:58.755163 9376 tablet_bootstrap.cc:654] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.755193 9375 tablet_bootstrap.cc:654] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.755303 9295 tablet_service.cc:1467] Processing CreateTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 (DEFAULT_TABLE table=test-table [id=42cb57346f9b403bad80ba93c70e0daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:20:58.755520 9376 log.cc:824] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.755586 9375 log.cc:824] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.755656 9295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4816e0f9634843e491f6f99ef6b77e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:58.756295 9375 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: No bootstrap required, opened a new log
I20260925 07:20:58.756366 9375 ts_tablet_manager.cc:1397] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:58.758989 9379 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Bootstrap starting.
I20260925 07:20:58.758996 9375 raft_consensus.cc:348] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.759162 9376 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: No bootstrap required, opened a new log
I20260925 07:20:58.759178 9375 raft_consensus.cc:374] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.759214 9375 raft_consensus.cc:729] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a19950ae53e4732b4e54f2b1b7e0438, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.759220 9376 ts_tablet_manager.cc:1397] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 07:20:58.759334 9375 consensus_queue.cc:260] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.759594 9375 ts_tablet_manager.cc:1428] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 07:20:58.759614 9099 heartbeater.cc:503] Master 127.4.47.62:45635 was elected leader, sending a full tablet report...
I20260925 07:20:58.759713 9379 tablet_bootstrap.cc:654] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:58.760000 9379 log.cc:824] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:58.760673 9379 tablet_bootstrap.cc:492] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: No bootstrap required, opened a new log
I20260925 07:20:58.760742 9379 ts_tablet_manager.cc:1397] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:20:58.761660 9376 raft_consensus.cc:348] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.761777 9376 raft_consensus.cc:374] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.761797 9376 raft_consensus.cc:729] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c986e07fb4f48b89c26208f4c8966ab, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.761914 9376 consensus_queue.cc:260] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.762187 9376 ts_tablet_manager.cc:1428] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 07:20:58.762226 9230 heartbeater.cc:503] Master 127.4.47.62:45635 was elected leader, sending a full tablet report...
I20260925 07:20:58.762840 9379 raft_consensus.cc:348] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.763013 9379 raft_consensus.cc:374] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:58.763052 9379 raft_consensus.cc:729] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dcb977b18a407195d677189995a93d, State: Initialized, Role: FOLLOWER
I20260925 07:20:58.763195 9379 consensus_queue.cc:260] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:58.763463 9379 ts_tablet_manager.cc:1428] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 07:20:58.763489 9361 heartbeater.cc:503] Master 127.4.47.62:45635 was elected leader, sending a full tablet report...
W20260925 07:20:58.868892 9231 tablet.cc:2369] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:20:58.986385 9362 tablet.cc:2369] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:20:59.005484 9100 tablet.cc:2369] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:20:59.038395 9383 raft_consensus.cc:482] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:20:59.038586 9383 raft_consensus.cc:504] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:59.039039 9383 leader_election.cc:288] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a19950ae53e4732b4e54f2b1b7e0438 (127.4.47.1:39495), 8c986e07fb4f48b89c26208f4c8966ab (127.4.47.2:38583)
I20260925 07:20:59.040416 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.039236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38583 (local address 127.4.47.3:45455)
0925 07:20:59.039297 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.039309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:59.039351 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:59.039532 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:59.039533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:59.039535 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:59.040035 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:59.040035 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:59.040044 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:59.040057 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:59.040256 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:59.040257 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:59.040258 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:20:59.040319 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:20:59.040332 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 07:20:59.040475 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.039363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:45455 (local address 127.4.47.2:38583)
0925 07:20:59.039411 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.039413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.039420 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:20:59.039451 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.039452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.039457 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.039527 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.040148 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.040150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.040150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.040155 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.040175 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.040181 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.040247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.040335 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.040348 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:20:59.040560 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.039385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:52899 (local address 127.4.47.1:39495)
0925 07:20:59.039433 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.039435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.039511 (+ 76us) server_negotiation.cc:407] Connection header received
0925 07:20:59.039575 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.039576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.039581 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.039633 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.040305 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.040307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.040307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.040312 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.040330 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.040336 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.040391 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.040462 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.040476 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:20:59.040570 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.039254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:39495 (local address 127.4.47.3:52899)
0925 07:20:59.039439 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.039448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:59.039509 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:59.039635 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:59.039636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:59.039638 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:20:59.040205 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:20:59.040206 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:20:59.040213 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:20:59.040224 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:20:59.040401 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:20:59.040402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:20:59.040402 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 07:20:59.040454 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:20:59.040467 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":17,"thread_start_us":96,"threads_started":1}
I20260925 07:20:59.040776 9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4816e0f9634843e491f6f99ef6b77e95" candidate_uuid: "97dcb977b18a407195d677189995a93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c986e07fb4f48b89c26208f4c8966ab" is_pre_election: true
I20260925 07:20:59.040851 9053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4816e0f9634843e491f6f99ef6b77e95" candidate_uuid: "97dcb977b18a407195d677189995a93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" is_pre_election: true
I20260925 07:20:59.040951 9184 raft_consensus.cc:2454] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dcb977b18a407195d677189995a93d in term 0.
I20260925 07:20:59.040988 9053 raft_consensus.cc:2454] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97dcb977b18a407195d677189995a93d in term 0.
I20260925 07:20:59.041210 9250 leader_election.cc:302] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c986e07fb4f48b89c26208f4c8966ab, 97dcb977b18a407195d677189995a93d; no voters:
I20260925 07:20:59.041353 9383 raft_consensus.cc:2781] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:20:59.041409 9383 raft_consensus.cc:482] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:20:59.041445 9383 raft_consensus.cc:3037] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.042132 9383 raft_consensus.cc:504] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:59.042289 9383 leader_election.cc:288] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [CANDIDATE]: Term 1 election: Requested vote from peers 4a19950ae53e4732b4e54f2b1b7e0438 (127.4.47.1:39495), 8c986e07fb4f48b89c26208f4c8966ab (127.4.47.2:38583)
I20260925 07:20:59.042464 9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4816e0f9634843e491f6f99ef6b77e95" candidate_uuid: "97dcb977b18a407195d677189995a93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c986e07fb4f48b89c26208f4c8966ab"
I20260925 07:20:59.042491 9053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4816e0f9634843e491f6f99ef6b77e95" candidate_uuid: "97dcb977b18a407195d677189995a93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a19950ae53e4732b4e54f2b1b7e0438"
I20260925 07:20:59.042564 9184 raft_consensus.cc:3037] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.042559 9053 raft_consensus.cc:3037] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.043370 9184 raft_consensus.cc:2454] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dcb977b18a407195d677189995a93d in term 1.
I20260925 07:20:59.043442 9053 raft_consensus.cc:2454] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97dcb977b18a407195d677189995a93d in term 1.
I20260925 07:20:59.043537 9250 leader_election.cc:302] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c986e07fb4f48b89c26208f4c8966ab, 97dcb977b18a407195d677189995a93d; no voters:
I20260925 07:20:59.043625 9383 raft_consensus.cc:2781] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:59.043819 9383 raft_consensus.cc:686] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 1 LEADER]: Becoming Leader. State: Replica: 97dcb977b18a407195d677189995a93d, State: Running, Role: LEADER
I20260925 07:20:59.043941 9383 consensus_queue.cc:237] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } }
I20260925 07:20:59.044868 8911 catalog_manager.cc:5495] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 97dcb977b18a407195d677189995a93d (127.4.47.3). New cstate: current_term: 1 leader_uuid: "97dcb977b18a407195d677189995a93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dcb977b18a407195d677189995a93d" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 35279 } health_report { overall_health: HEALTHY } } }
I20260925 07:20:59.112150 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.111119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.4.47.3:35279)
0925 07:20:59.111256 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.111258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.111268 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:20:59.111310 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.111311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.111323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.111375 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.111911 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.111912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.111912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.111916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.111929 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.111933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.111980 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.112023 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.112034 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:20:59.114048 9053 raft_consensus.cc:1261] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 1 FOLLOWER]: Refusing update from remote peer 97dcb977b18a407195d677189995a93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:59.114048 9184 raft_consensus.cc:1261] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 1 FOLLOWER]: Refusing update from remote peer 97dcb977b18a407195d677189995a93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:20:59.114420 9385 consensus_queue.cc:1035] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a19950ae53e4732b4e54f2b1b7e0438" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 39495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:59.114567 9383 consensus_queue.cc:1035] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c986e07fb4f48b89c26208f4c8966ab" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:20:59.117022 9388 mvcc.cc:205] Tried to move back new op lower bound from 7333154238928138240 to 7333154238644441088. Current Snapshot: MvccSnapshot[applied={T|T < 7333154238928138240}]
I20260925 07:20:59.117159 9389 mvcc.cc:205] Tried to move back new op lower bound from 7333154238928138240 to 7333154238644441088. Current Snapshot: MvccSnapshot[applied={T|T < 7333154238928138240}]
I20260925 07:20:59.120024 8910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:59.120136 8910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:20:59.120937 8910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4ab0c7144874dc1b7b65a85e9438978: Sending DeleteTablet for 3 replicas of tablet 4816e0f9634843e491f6f99ef6b77e95
I20260925 07:20:59.121325 9295 tablet_service.cc:1514] Processing DeleteTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:59 UTC) from {username='slave'} at 127.0.0.1:38528
I20260925 07:20:59.121347 9164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:59 UTC) from {username='slave'} at 127.0.0.1:43222
I20260925 07:20:59.121366 9033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4816e0f9634843e491f6f99ef6b77e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:20:59 UTC) from {username='slave'} at 127.0.0.1:43442
I20260925 07:20:59.121623 9400 tablet_replica.cc:330] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438: stopping tablet replica
I20260925 07:20:59.121727 9400 raft_consensus.cc:2229] T 4816e0f9634843e491f6f99ef6b77e95 P 4a19950ae53e4732b4e54f2b1b7e0438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:59.121855 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8971
I20260925 07:20:59.123176 9398 tablet_replica.cc:330] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: stopping tablet replica
I20260925 07:20:59.123319 9398 raft_consensus.cc:2229] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:20:59.123514 9398 raft_consensus.cc:2258] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:59.124019 9398 ts_tablet_manager.cc:1905] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:20:59.126183 9398 ts_tablet_manager.cc:1918] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:20:59.126312 9398 log.cc:1196] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/4816e0f9634843e491f6f99ef6b77e95
I20260925 07:20:59.126672 9398 ts_tablet_manager.cc:1939] T 4816e0f9634843e491f6f99ef6b77e95 P 97dcb977b18a407195d677189995a93d: Deleting consensus metadata
I20260925 07:20:59.127204 8895 catalog_manager.cc:4841] TS 97dcb977b18a407195d677189995a93d (127.4.47.3:35279): tablet 4816e0f9634843e491f6f99ef6b77e95 (table test-table [id=42cb57346f9b403bad80ba93c70e0daf]) successfully deleted
I20260925 07:20:59.127382 9399 tablet_replica.cc:330] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: stopping tablet replica
I20260925 07:20:59.127497 9399 raft_consensus.cc:2229] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:20:59.127612 9399 raft_consensus.cc:2258] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:20:59.128037 9399 ts_tablet_manager.cc:1905] T 4816e0f9634843e491f6f99ef6b77e95 P 8c986e07fb4f48b89c26208f4c8966ab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:20:59.128455 8896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:39495 (error 108)
I20260925 07:20:59.128701 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9102
I20260925 07:20:59.130570 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.130439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:39495 (local address 127.0.0.1:43454)
0925 07:20:59.130491 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.130533 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:39495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:20:59.130686 8896 catalog_manager.cc:4584] TS 4a19950ae53e4732b4e54f2b1b7e0438 (127.4.47.1:39495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4816e0f9634843e491f6f99ef6b77e95: Network error: Client connection negotiation failed: client connection to 127.4.47.1:39495: connect: Connection refused (error 111)
I20260925 07:20:59.133701 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9233
I20260925 07:20:59.134467 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.133789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:38583 (local address 127.0.0.1:43234)
0925 07:20:59.134395 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.134429 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:38583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585}
I20260925 07:20:59.138481 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 8880
2026-09-25T07:20:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (921 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 07:20:59.158267 4284 mini_oidc.cc:152] Starting JWKS server
I20260925 07:20:59.159416 4284 webserver.cc:460] Webserver started at https://127.0.0.1:33895/ using document root <none> and password file <none>
I20260925 07:20:59.159521 4284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:20:59.159686 4284 webserver.cc:460] Webserver started at http://127.0.0.1:45799/ using document root <none> and password file <none>
2026-09-25T07:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:59Z Disabled control of system clock
I20260925 07:20:59.167449 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38079
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38079
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/testchainca.cert with env {}
W20260925 07:20:59.248417 9407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:59.248641 9407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:59.248662 9407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:59.249810 9407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:59.249848 9407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:59.249861 9407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:59.249874 9407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:59.249886 9407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:20:59.249898 9407 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:20:59.249910 9407 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33895/jwks.json
I20260925 07:20:59.251444 9407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:59.251647 9407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:59.251809 9407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:59.256760 9415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.256767 9413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.256773 9412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:59.257071 9407 server_base.cc:1034] running on GCE node
I20260925 07:20:59.257355 9407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:59.257609 9407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:59.258756 9407 hybrid_clock.cc:648] HybridClock initialized: now 1790320859258729 us; error 43 us; skew 500 ppm
I20260925 07:20:59.260094 9407 webserver.cc:460] Webserver started at http://127.4.47.62:43915/ using document root <none> and password file <none>
I20260925 07:20:59.260375 9407 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:59.260424 9407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:59.260512 9407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:59.261652 9407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "47feee7141344ba58107e6b96e697090"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "b2e83d6a346d25eb8588c474dd293c3e"
server_key_iv: "fdf3299992e9e9fce73290ce9778356f"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.262092 9407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "47feee7141344ba58107e6b96e697090"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "b2e83d6a346d25eb8588c474dd293c3e"
server_key_iv: "fdf3299992e9e9fce73290ce9778356f"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.263893 9407 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 07:20:59.264979 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:59.265468 9407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:59.265553 9407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "47feee7141344ba58107e6b96e697090"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "b2e83d6a346d25eb8588c474dd293c3e"
server_key_iv: "fdf3299992e9e9fce73290ce9778356f"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.265626 9407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:59.289639 9407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:59.289938 9407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:59.290045 9407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:59.293897 9407 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:38079
I20260925 07:20:59.294013 9473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:38079 every 8 connection(s)
I20260925 07:20:59.294379 9407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:59.295514 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:59.297758 9474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: Bootstrap starting.
I20260925 07:20:59.298326 9474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:59.298568 9474 log.cc:824] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:59.299441 9474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: No bootstrap required, opened a new log
I20260925 07:20:59.300982 9474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } }
I20260925 07:20:59.301127 9474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:59.301151 9474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47feee7141344ba58107e6b96e697090, State: Initialized, Role: FOLLOWER
I20260925 07:20:59.301287 9474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } }
I20260925 07:20:59.301374 9474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:59.301411 9474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:59.301463 9474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.301641 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9407
I20260925 07:20:59.301724 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:59.301905 4284 external_mini_cluster.cc:1428] Setting key 98c217401e470fc1afa2ee5ef7031614
I20260925 07:20:59.302091 9474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } }
I20260925 07:20:59.302213 9474 leader_election.cc:302] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47feee7141344ba58107e6b96e697090; no voters:
I20260925 07:20:59.302388 9474 leader_election.cc:288] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:59.302632 9474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:59.303145 9480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:59.303560 9480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [term 1 LEADER]: Becoming Leader. State: Replica: 47feee7141344ba58107e6b96e697090, State: Running, Role: LEADER
I20260925 07:20:59.303653 9480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } }
I20260925 07:20:59.303974 9480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47feee7141344ba58107e6b96e697090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } } }
I20260925 07:20:59.304028 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47feee7141344ba58107e6b96e697090. Latest consensus state: current_term: 1 leader_uuid: "47feee7141344ba58107e6b96e697090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feee7141344ba58107e6b96e697090" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38079 } } }
I20260925 07:20:59.304127 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:59.304042 9480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090 [sys.catalog]: This master's current role is: LEADER
W20260925 07:20:59.304693 9493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:20:59.304746 9493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:20:59.304803 9494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:59.304937 9494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:59.306588 9494 catalog_manager.cc:1347] Generated new cluster ID: 6ddc8808bc0a4478b9023bb12a1f6ab4
I20260925 07:20:59.306645 9494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:59.308228 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.302366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.4.47.62:38079)
0925 07:20:59.302576 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.302585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.302619 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:20:59.302739 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.302745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.302824 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.302970 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.303202 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.304659 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.305946 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.306471 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.306611 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.307538 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.307542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.307542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.307727 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.307759 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.307770 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.307857 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.307930 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.308014 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260925 07:20:59.309089 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:20:59.333117 9494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:59.333868 9494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:59.342710 9494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47feee7141344ba58107e6b96e697090: Generated new TSK 0
I20260925 07:20:59.342945 9494 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 07:20:59.378597 9504 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 07:20:59.378698 9504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 07:20:59.378849 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.356243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.4.47.62:38079)
0925 07:20:59.356305 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.356307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.356461 (+ 154us) server_negotiation.cc:407] Connection header received
0925 07:20:59.356570 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.356571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.356582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.356641 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:20:59.356814 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.357269 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.358425 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.358599 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.358648 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.358870 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:20:59.378578 (+ 19708us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 07:20:59.378762 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 07:20:59.378907 9479 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 07:20:59.379253 9503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:20:59.356167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38079 (local address 127.0.0.1:45226)
0925 07:20:59.356383 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.356396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:59.356476 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:59.356668 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:59.356669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:59.356671 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:20:59.356751 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.356753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.357375 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.357376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.358337 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.358340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.358668 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.358672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.358740 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.358763 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:20:59.378745 (+ 19982us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 07:20:59.378877 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.47.62:38079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.47.62:38079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 07:20:59.379753 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9407
2026-09-25T07:20:59Z chronyd exiting
I20260925 07:20:59.416013 4284 test_util.cc:175] -----------------------------------------------
I20260925 07:20:59.416092 4284 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790320837143632-4284-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 07:20:59.416826 4284 mini_oidc.cc:152] Starting JWKS server
I20260925 07:20:59.417483 4284 webserver.cc:460] Webserver started at https://127.0.0.1:43215/ using document root <none> and password file <none>
I20260925 07:20:59.417521 4284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:20:59.417658 4284 webserver.cc:460] Webserver started at http://127.0.0.1:33329/ using document root <none> and password file <none>
2026-09-25T07:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:59Z Disabled control of system clock
I20260925 07:20:59.425199 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45539
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/testchainca.cert with env {}
W20260925 07:20:59.504994 9510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:59.505251 9510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:59.505290 9510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:59.506449 9510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:59.506513 9510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:59.506534 9510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:59.506554 9510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:59.506582 9510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:20:59.506605 9510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:20:59.506634 9510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43215/jwks.json
I20260925 07:20:59.508154 9510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:59.508440 9510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:59.508654 9510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:59.513545 9516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.513551 9515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:59.513595 9510 server_base.cc:1034] running on GCE node
W20260925 07:20:59.513607 9518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:59.513953 9510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:59.514261 9510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:59.515440 9510 hybrid_clock.cc:648] HybridClock initialized: now 1790320859515412 us; error 39 us; skew 500 ppm
I20260925 07:20:59.521498 9510 webserver.cc:460] Webserver started at http://127.4.47.62:33709/ using document root <none> and password file <none>
I20260925 07:20:59.521768 9510 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:59.521858 9510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:59.522011 9510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:59.523155 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "805d1cb362194979bdafb61059e33bc3"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "867dfaf6a7990a8370625460ca7aa633"
server_key_iv: "1d6bc48d966ec1300456ec2c6eaf0f89"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.523528 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "805d1cb362194979bdafb61059e33bc3"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "867dfaf6a7990a8370625460ca7aa633"
server_key_iv: "1d6bc48d966ec1300456ec2c6eaf0f89"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.524863 9510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:20:59.525645 9524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:59.525825 9510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:20:59.525931 9510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "805d1cb362194979bdafb61059e33bc3"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "867dfaf6a7990a8370625460ca7aa633"
server_key_iv: "1d6bc48d966ec1300456ec2c6eaf0f89"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.526006 9510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:59.546983 9510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:59.547366 9510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:59.547529 9510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:59.551570 9510 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:45539
I20260925 07:20:59.551610 9576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:45539 every 8 connection(s)
I20260925 07:20:59.551987 9510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:59.552918 9577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:59.555256 9577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3: Bootstrap starting.
I20260925 07:20:59.555785 9577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:59.556015 9577 log.cc:824] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:59.556643 9577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3: No bootstrap required, opened a new log
I20260925 07:20:59.558235 9577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } }
I20260925 07:20:59.558373 9577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:59.558396 9577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805d1cb362194979bdafb61059e33bc3, State: Initialized, Role: FOLLOWER
I20260925 07:20:59.558501 9577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } }
I20260925 07:20:59.558552 9577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:59.558573 9577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:59.558598 9577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.559059 9577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } }
I20260925 07:20:59.559237 9577 leader_election.cc:302] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 805d1cb362194979bdafb61059e33bc3; no voters:
I20260925 07:20:59.559396 9577 leader_election.cc:288] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:59.559432 9580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:59.559553 9580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 805d1cb362194979bdafb61059e33bc3, State: Running, Role: LEADER
I20260925 07:20:59.559670 9577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:59.559665 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9510
I20260925 07:20:59.559640 9580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } }
I20260925 07:20:59.559762 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:59.559937 4284 external_mini_cluster.cc:1428] Setting key ac57d0dc8db320a95a487e4ae0508c19
I20260925 07:20:59.559957 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805d1cb362194979bdafb61059e33bc3. Latest consensus state: current_term: 1 leader_uuid: "805d1cb362194979bdafb61059e33bc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } } }
I20260925 07:20:59.560029 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:59.559969 9581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "805d1cb362194979bdafb61059e33bc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805d1cb362194979bdafb61059e33bc3" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45539 } } }
I20260925 07:20:59.560071 9581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3 [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:59.560642 9588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:59.561219 9588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:59.562906 9588 catalog_manager.cc:1347] Generated new cluster ID: fdf9856562db42cea14567890336cd96
I20260925 07:20:59.562960 9588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:59.565739 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.560392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.4.47.62:45539)
0925 07:20:59.560676 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.560687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.560948 (+ 261us) server_negotiation.cc:407] Connection header received
0925 07:20:59.561201 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.561210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.561287 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.561447 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.561777 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.563054 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.563875 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.564296 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.564440 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.565091 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.565094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.565095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.565268 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.565300 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.565310 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.565386 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.565458 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.565531 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260925 07:20:59.566612 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:20:59.569433 9588 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:59.570050 9588 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:59.583385 9588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 805d1cb362194979bdafb61059e33bc3: Generated new TSK 0
I20260925 07:20:59.583612 9588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 07:20:59.606292 9607 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 07:20:59.606387 9607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 07:20:59.606747 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.589789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.4.47.62:45539)
0925 07:20:59.589834 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.589837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.589945 (+ 108us) server_negotiation.cc:407] Connection header received
0925 07:20:59.590019 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.590020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.590030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.590091 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:20:59.590228 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.590736 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.591870 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.592070 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.592123 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.592312 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:20:59.606559 (+ 14247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 07:20:59.606692 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56938: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:20:59.606818 9589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56938: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 07:20:59.606951 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:20:59.589688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45539 (local address 127.0.0.1:56938)
0925 07:20:59.589857 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.589879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:59.589949 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:59.590106 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:59.590107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:59.590109 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:20:59.590168 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.590170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.590813 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.590814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.591787 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.591790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.592142 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.592145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.592213 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.592219 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:20:59.606690 (+ 14471us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 07:20:59.606838 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.47.62:45539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260925 07:20:59.607187 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9510
2026-09-25T07:20:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 07:20:59.644922 4284 mini_oidc.cc:152] Starting JWKS server
I20260925 07:20:59.645571 4284 webserver.cc:460] Webserver started at https://127.0.0.1:36863/ using document root <none> and password file <none>
I20260925 07:20:59.645612 4284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:20:59.645749 4284 webserver.cc:460] Webserver started at http://127.0.0.1:45235/ using document root <none> and password file <none>
2026-09-25T07:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:59Z Disabled control of system clock
I20260925 07:20:59.653738 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:46781
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:46781
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 07:20:59.735879 9613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:59.736088 9613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:59.736109 9613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:59.737200 9613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:59.737234 9613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:59.737247 9613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:59.737259 9613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:59.737277 9613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:20:59.737296 9613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:20:59.737308 9613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36863/jwks.json
I20260925 07:20:59.738727 9613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:46781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:59.739006 9613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:59.739338 9613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:59.744179 9619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.744220 9621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.744179 9618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:59.744503 9613 server_base.cc:1034] running on GCE node
I20260925 07:20:59.744774 9613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:59.745048 9613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:59.746189 9613 hybrid_clock.cc:648] HybridClock initialized: now 1790320859746164 us; error 37 us; skew 500 ppm
I20260925 07:20:59.754231 9613 webserver.cc:460] Webserver started at http://127.4.47.62:44489/ using document root <none> and password file <none>
I20260925 07:20:59.754454 9613 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:59.754498 9613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:59.754593 9613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:59.755719 9613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "0348c977794d4912b25b3c8f246379fc"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "c5b81a83eba9b0eb93e4b00447e7a931"
server_key_iv: "60d639a686870932300e3074857512d7"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.756033 9613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "0348c977794d4912b25b3c8f246379fc"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "c5b81a83eba9b0eb93e4b00447e7a931"
server_key_iv: "60d639a686870932300e3074857512d7"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.757316 9613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:59.758008 9627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:59.758219 9613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:20:59.758342 9613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "0348c977794d4912b25b3c8f246379fc"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "c5b81a83eba9b0eb93e4b00447e7a931"
server_key_iv: "60d639a686870932300e3074857512d7"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.758431 9613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:20:59.794056 9613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:20:59.794418 9613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:20:59.794588 9613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:20:59.798641 9613 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:46781
I20260925 07:20:59.798686 9679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:46781 every 8 connection(s)
I20260925 07:20:59.799077 9613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:20:59.799780 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:20:59.802393 9680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc: Bootstrap starting.
I20260925 07:20:59.802977 9680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc: Neither blocks nor log segments found. Creating new log.
I20260925 07:20:59.803313 9680 log.cc:824] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:20:59.803997 9680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc: No bootstrap required, opened a new log
I20260925 07:20:59.805601 9680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } }
I20260925 07:20:59.805742 9680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:20:59.805764 9680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0348c977794d4912b25b3c8f246379fc, State: Initialized, Role: FOLLOWER
I20260925 07:20:59.805943 9680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } }
I20260925 07:20:59.806025 9680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:20:59.806066 9680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:20:59.806118 9680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:20:59.806648 9680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } }
I20260925 07:20:59.806759 9680 leader_election.cc:302] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0348c977794d4912b25b3c8f246379fc; no voters:
I20260925 07:20:59.806936 9680 leader_election.cc:288] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:20:59.807020 9683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:20:59.807253 9680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:20:59.807307 9683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [term 1 LEADER]: Becoming Leader. State: Replica: 0348c977794d4912b25b3c8f246379fc, State: Running, Role: LEADER
I20260925 07:20:59.807410 9683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } }
I20260925 07:20:59.807837 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0348c977794d4912b25b3c8f246379fc. Latest consensus state: current_term: 1 leader_uuid: "0348c977794d4912b25b3c8f246379fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } } }
I20260925 07:20:59.807921 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:59.807969 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0348c977794d4912b25b3c8f246379fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0348c977794d4912b25b3c8f246379fc" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46781 } } }
I20260925 07:20:59.808038 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc [sys.catalog]: This master's current role is: LEADER
I20260925 07:20:59.808151 9692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:20:59.808342 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9613
I20260925 07:20:59.808455 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:20:59.808640 4284 external_mini_cluster.cc:1428] Setting key ef9230a9c1839ac1b9ce9a2e6dcd831b
I20260925 07:20:59.808713 9692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:20:59.810345 9692 catalog_manager.cc:1347] Generated new cluster ID: 766543e0eaac454f96f992c17fe70879
I20260925 07:20:59.810398 9692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:20:59.814730 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.809266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.4.47.62:46781)
0925 07:20:59.809477 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.809484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.809507 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:20:59.809595 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.809600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.809659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.809838 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:20:59.810066 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.811450 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.812579 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.813162 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.813280 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.814018 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:20:59.814021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:20:59.814022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:20:59.814235 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:20:59.814275 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:20:59.814288 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:20:59.814348 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:20:59.814445 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:20:59.814517 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260925 07:20:59.815565 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:20:59.822558 9692 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:20:59.823058 9692 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:20:59.831130 9692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0348c977794d4912b25b3c8f246379fc: Generated new TSK 0
I20260925 07:20:59.831454 9692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:20:59.851604 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:20:59.847333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.4.47.62:46781)
0925 07:20:59.847383 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:20:59.847385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:20:59.847568 (+ 183us) server_negotiation.cc:407] Connection header received
0925 07:20:59.847693 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:20:59.847694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:20:59.847707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:20:59.847759 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:20:59.847988 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.848604 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.849768 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.850031 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.850104 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.850346 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:20:59.851362 (+ 1016us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 07:20:59.851521 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44522: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 07:20:59.851672 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:20:59.847279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:46781 (local address 127.0.0.1:44522)
0925 07:20:59.847503 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 07:20:59.847514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:20:59.847599 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:20:59.847836 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:20:59.847837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:20:59.847839 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:20:59.847901 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.847903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.848690 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.848691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.849677 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:20:59.849680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:20:59.850121 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:20:59.850125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:20:59.850209 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:20:59.850216 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:20:59.851469 (+ 1253us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 07:20:59.851562 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.47.62:46781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260925 07:20:59.851679 9702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44522: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36863/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 07:20:59.852103 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9613
E20260925 07:20:59.868271 9710 webserver.cc:533] Webserver: sslize failed:
E20260925 07:20:59.868371 9710 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T07:20:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T07:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:20:59Z Disabled control of system clock
I20260925 07:20:59.897578 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36655
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:36655
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/testchainca.cert with env {}
W20260925 07:20:59.977835 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:20:59.978098 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:20:59.978135 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:20:59.979343 9714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:20:59.979413 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:20:59.979432 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:20:59.979452 9714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:20:59.979483 9714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:20:59.979509 9714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:20:59.979534 9714 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 07:20:59.981048 9714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:36655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:20:59.981355 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:20:59.981578 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:20:59.986389 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.986399 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:20:59.986389 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:20:59.986670 9714 server_base.cc:1034] running on GCE node
I20260925 07:20:59.986917 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:20:59.987273 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:20:59.988478 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790320859988451 us; error 31 us; skew 500 ppm
I20260925 07:20:59.989867 9714 webserver.cc:460] Webserver started at http://127.4.47.62:44819/ using document root <none> and password file <none>
I20260925 07:20:59.990118 9714 fs_manager.cc:359] Metadata directory not provided
I20260925 07:20:59.990187 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:20:59.990343 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:20:59.991557 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "2218d56989364f04ba2c4f0b482d031d"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "e3f870d1a8017d108bfa810f653d046a"
server_key_iv: "397fd1c682507e377958ff95d5ff5883"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.991928 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "2218d56989364f04ba2c4f0b482d031d"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "e3f870d1a8017d108bfa810f653d046a"
server_key_iv: "397fd1c682507e377958ff95d5ff5883"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.993197 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:20:59.994038 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:20:59.994357 9714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:20:59.994462 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "2218d56989364f04ba2c4f0b482d031d"
format_stamp: "Formatted at 2026-09-25 07:20:59 on dist-test-slave-zw07"
server_key: "e3f870d1a8017d108bfa810f653d046a"
server_key_iv: "397fd1c682507e377958ff95d5ff5883"
server_key_version: "encryptionkey@0"
I20260925 07:20:59.994544 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:00.001886 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:00.002236 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:00.002384 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:00.006464 9714 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:36655
I20260925 07:21:00.006498 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:36655 every 8 connection(s)
I20260925 07:21:00.006932 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:00.008008 9781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:00.010396 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d: Bootstrap starting.
I20260925 07:21:00.011008 9781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:00.011376 9781 log.cc:824] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:00.011533 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9714
I20260925 07:21:00.011659 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:00.011830 4284 external_mini_cluster.cc:1428] Setting key c9d25afb822b573aa1d0ab254f172e40
I20260925 07:21:00.012212 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d: No bootstrap required, opened a new log
I20260925 07:21:00.015100 9781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } }
I20260925 07:21:00.015326 9781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:00.015358 9781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2218d56989364f04ba2c4f0b482d031d, State: Initialized, Role: FOLLOWER
I20260925 07:21:00.015489 9781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } }
I20260925 07:21:00.015578 9781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:00.015621 9781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:00.015671 9781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:00.016284 9781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } }
I20260925 07:21:00.016389 9781 leader_election.cc:302] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2218d56989364f04ba2c4f0b482d031d; no voters:
I20260925 07:21:00.016564 9781 leader_election.cc:288] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:00.016649 9786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:00.016820 9786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [term 1 LEADER]: Becoming Leader. State: Replica: 2218d56989364f04ba2c4f0b482d031d, State: Running, Role: LEADER
I20260925 07:21:00.016829 9781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:00.016956 9786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } }
I20260925 07:21:00.017341 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2218d56989364f04ba2c4f0b482d031d. Latest consensus state: current_term: 1 leader_uuid: "2218d56989364f04ba2c4f0b482d031d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } } }
I20260925 07:21:00.017490 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2218d56989364f04ba2c4f0b482d031d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2218d56989364f04ba2c4f0b482d031d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36655 } } }
I20260925 07:21:00.017546 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:00.017417 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:00.017872 9795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:00.018469 9795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:00.019409 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:00.012361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.4.47.62:36655)
0925 07:21:00.012604 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:00.012615 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:00.012652 (+ 37us) server_negotiation.cc:407] Connection header received
0925 07:21:00.013528 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:00.013538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:00.013620 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:00.013791 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:00.013950 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.015452 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.016278 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.016927 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.017416 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:00.018595 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:00.018599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:00.018599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:00.018831 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:00.018874 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:00.018888 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:00.018957 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:00.019090 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 07:21:00.019177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260925 07:21:00.020385 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:21:00.020521 9795 catalog_manager.cc:1347] Generated new cluster ID: 3a063674564c47bea40fc1ee340e7869
I20260925 07:21:00.020573 9795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:00.055732 9795 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:00.056324 9795 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:00.065270 9795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2218d56989364f04ba2c4f0b482d031d: Generated new TSK 0
I20260925 07:21:00.065526 9795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:21:00.115684 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:00.073057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.4.47.62:36655)
0925 07:21:00.073119 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:00.073121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:00.073233 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:21:00.073315 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:00.073316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:00.073331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:00.073413 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:21:00.073550 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.074044 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.074849 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.075045 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.075156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:00.075355 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:21:00.115407 (+ 40052us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 07:21:00.115604 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42820: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 07:21:00.115765 9785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42820: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 07:21:00.115794 9810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:00.072992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36655 (local address 127.0.0.1:42820)
0925 07:21:00.073163 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:00.073177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:00.073252 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:00.073410 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:00.073411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:00.073413 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:21:00.073473 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:00.073476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.074145 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.074147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:00.074747 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:00.074749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:00.075169 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:00.075173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:00.075236 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:00.075241 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:21:00.115570 (+ 40329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 07:21:00.115689 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.47.62:36655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 07:21:00.116145 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9714
2026-09-25T07:21:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (243 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:00 dist-test-slave-zw07 krb5kdc[9822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:00 dist-test-slave-zw07 krb5kdc[9822](info): set up 2 sockets
Sep 25 07:21:00 dist-test-slave-zw07 krb5kdc[9822](info): commencing operation
krb5kdc: starting...
W20260925 07:21:01.323272 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:21:01.442219 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33207
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:01.521975 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:01.522189 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:01.522212 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:01.523430 9835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:01.523474 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:01.523489 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:01.523502 9835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:01.523514 9835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:01.524986 9835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:01.525233 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:01.525460 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:01.530398 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:01.530405 9841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:01.530405 9840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:01.530625 9835 server_base.cc:1034] running on GCE node
I20260925 07:21:01.530831 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:01.531119 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:01.532285 9835 hybrid_clock.cc:648] HybridClock initialized: now 1790320861532268 us; error 35 us; skew 500 ppm
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:01.534744 9835 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:21:01.535187 9835 webserver.cc:460] Webserver started at http://127.4.47.62:34321/ using document root <none> and password file <none>
I20260925 07:21:01.535372 9835 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:01.535419 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:01.535508 9835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:01.536511 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "53291963cf9c40aa9407b851441e03af"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "91314515269efbfcc04da249503fe7f1"
server_key_iv: "d177d97bdb631fe692035219f5b36bcf"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.536811 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "53291963cf9c40aa9407b851441e03af"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "91314515269efbfcc04da249503fe7f1"
server_key_iv: "d177d97bdb631fe692035219f5b36bcf"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.538061 9835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:01.538780 9850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.538956 9835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:01.539059 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "53291963cf9c40aa9407b851441e03af"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "91314515269efbfcc04da249503fe7f1"
server_key_iv: "d177d97bdb631fe692035219f5b36bcf"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.539182 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:01.557184 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:01.559202 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:01.559368 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:01.563228 9835 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33207
I20260925 07:21:01.563283 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33207 every 8 connection(s)
I20260925 07:21:01.563663 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:01.564513 9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:01.566694 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af: Bootstrap starting.
I20260925 07:21:01.566982 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9835
I20260925 07:21:01.567121 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:01.567309 4284 external_mini_cluster.cc:1428] Setting key bb1b6f3f0cb4d1d6ea6788637a15cddb
I20260925 07:21:01.567484 9903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:01.567848 9903 log.cc:824] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:01.568532 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af: No bootstrap required, opened a new log
I20260925 07:21:01.570315 9903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } }
I20260925 07:21:01.570488 9903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:01.570537 9903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53291963cf9c40aa9407b851441e03af, State: Initialized, Role: FOLLOWER
I20260925 07:21:01.570688 9903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } }
I20260925 07:21:01.570778 9903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:01.570822 9903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:01.570870 9903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:01.571455 9903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } }
I20260925 07:21:01.571571 9903 leader_election.cc:302] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53291963cf9c40aa9407b851441e03af; no voters:
I20260925 07:21:01.571731 9903 leader_election.cc:288] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:01.571784 9908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:01.571938 9908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [term 1 LEADER]: Becoming Leader. State: Replica: 53291963cf9c40aa9407b851441e03af, State: Running, Role: LEADER
I20260925 07:21:01.571988 9903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:01.572036 9908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } }
I20260925 07:21:01.572366 9909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53291963cf9c40aa9407b851441e03af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } } }
I20260925 07:21:01.572362 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53291963cf9c40aa9407b851441e03af. Latest consensus state: current_term: 1 leader_uuid: "53291963cf9c40aa9407b851441e03af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53291963cf9c40aa9407b851441e03af" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33207 } } }
I20260925 07:21:01.572481 9909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:01.572499 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:01.572685 9912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:01.573180 9912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:01.575160 9925 mvcc.cc:205] Tried to move back new op lower bound from 7333154249007783936 to 7333154248999456768. Current Snapshot: MvccSnapshot[applied={T|T < 7333154249007783936}]
I20260925 07:21:01.575704 9912 catalog_manager.cc:1347] Generated new cluster ID: dfe8e840ee7445b6b5656295769cbdb8
I20260925 07:21:01.575754 9912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:01.578931 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:01.567863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.4.47.62:33207)
0925 07:21:01.568059 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:01.568076 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:01.568101 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:21:01.568503 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:01.568509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:01.568555 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:01.568732 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:01.569357 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.570236 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.571243 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.571754 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.572186 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:01.574995 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:01.574998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:01.574999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:01.575010 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:01.577027 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.577421 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.577424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.577424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.577521 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.577719 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.577720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.577720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.578024 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:01.578166 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:01.578579 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 07:21:01.578711 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260925 07:21:01.581974 9912 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:01.582458 9912 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:01.588231 9912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53291963cf9c40aa9407b851441e03af: Generated new TSK 0
I20260925 07:21:01.588402 9912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:21:01.617774 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:01.699644 9929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:01.699874 9929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:01.699900 9929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:01.701125 9929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:01.701169 9929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:01.701215 9929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:01.702826 9929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:01.703038 9929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:01.703279 9929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:01.708326 9934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:01.708401 9935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:01.708456 9929 server_base.cc:1034] running on GCE node
W20260925 07:21:01.708336 9937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:01.708799 9929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:01.709057 9929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:01.710249 9929 hybrid_clock.cc:648] HybridClock initialized: now 1790320861710244 us; error 34 us; skew 500 ppm
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:01.712810 9929 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:21:01.713248 9929 webserver.cc:460] Webserver started at http://127.4.47.1:34039/ using document root <none> and password file <none>
I20260925 07:21:01.713493 9929 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:01.713562 9929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:01.713740 9929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:01.714790 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "1eb382e731af48cea8cfb99170b758ad"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "075d13243b7dd0f9375c00c5a5468fec"
server_key_iv: "7fdcb2b8b69e092f74a28c50d3d9714d"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.715195 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "1eb382e731af48cea8cfb99170b758ad"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "075d13243b7dd0f9375c00c5a5468fec"
server_key_iv: "7fdcb2b8b69e092f74a28c50d3d9714d"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.716463 9929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:01.717139 9944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.717288 9929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:01.717391 9929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "1eb382e731af48cea8cfb99170b758ad"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "075d13243b7dd0f9375c00c5a5468fec"
server_key_iv: "7fdcb2b8b69e092f74a28c50d3d9714d"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.717468 9929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:01.746803 9929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:01.748886 9929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:01.749006 9929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:01.749181 9929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:01.749496 9929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:01.749529 9929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.749552 9929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:01.749599 9929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.755249 9929 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:33059
I20260925 07:21:01.755304 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:33059 every 8 connection(s)
I20260925 07:21:01.755613 9929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:01.763008 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 9929
I20260925 07:21:01.763155 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:01.763355 4284 external_mini_cluster.cc:1428] Setting key 2d77390e1157fad31d762aef8f6ca5c6
I20260925 07:21:01.765038 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:01.756343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:38915 (local address 127.4.47.62:33207)
0925 07:21:01.756416 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:01.756418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:01.756640 (+ 222us) server_negotiation.cc:407] Connection header received
0925 07:21:01.757551 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:01.757551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:01.757561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:01.757616 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:01.758216 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.758721 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.759741 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.759960 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.760016 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:01.762017 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:01.762018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:01.762019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:01.762025 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:01.763494 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.763993 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.763994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.763995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.764064 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.764319 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.764320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.764321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.764484 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:01.764556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:01.764798 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 07:21:01.764893 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:21:01.765053 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:01.756342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33207 (local address 127.4.47.1:38915)
0925 07:21:01.756510 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:01.756552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:01.757460 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:01.757681 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:01.757684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:01.757940 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:01.758137 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:01.758141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.758802 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.758803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:01.759664 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:01.759667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.760036 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.760037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:01.760136 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:01.760585 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:01.760593 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:01.761930 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:01.763577 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:01.763585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:01.763587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:01.763912 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:01.764127 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:01.764128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:01.764129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:01.764251 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:01.764551 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:01.764556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:01.764670 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 07:21:01.764754 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:21:01.764890 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260925 07:21:01.766067 10058 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33207
I20260925 07:21:01.766185 10058 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:01.766371 10058 heartbeater.cc:511] Master 127.4.47.62:33207 requested a full tablet report, sending...
I20260925 07:21:01.766918 9867 ts_manager.cc:194] Registered new tserver with Master: 1eb382e731af48cea8cfb99170b758ad (127.4.47.1:33059)
I20260925 07:21:01.767598 9867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:38915
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:21:01.789700 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:01.869390 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:01.869589 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:01.869613 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:01.870682 10063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:01.870715 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:01.870741 10063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:01.872242 10063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:01.872433 10063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:01.872629 10063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:01.877959 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:01.878047 10063 server_base.cc:1034] running on GCE node
W20260925 07:21:01.877959 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:01.877977 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:01.878396 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:01.878584 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:01.879735 10063 hybrid_clock.cc:648] HybridClock initialized: now 1790320861879728 us; error 30 us; skew 500 ppm
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:01.882253 10063 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:21:01.882694 10063 webserver.cc:460] Webserver started at http://127.4.47.2:39873/ using document root <none> and password file <none>
I20260925 07:21:01.882928 10063 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:01.882973 10063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:01.883162 10063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:01.884205 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "4754c33188e4441bbcd4428f8b82c36c"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "23eddf798a1dc96caa7369d2e93c0526"
server_key_iv: "e9aee6181ae7199c2ce20f8355442e61"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.884572 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "4754c33188e4441bbcd4428f8b82c36c"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "23eddf798a1dc96caa7369d2e93c0526"
server_key_iv: "e9aee6181ae7199c2ce20f8355442e61"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.885819 10063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:01.886586 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.886776 10063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:01.886868 10063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "4754c33188e4441bbcd4428f8b82c36c"
format_stamp: "Formatted at 2026-09-25 07:21:01 on dist-test-slave-zw07"
server_key: "23eddf798a1dc96caa7369d2e93c0526"
server_key_iv: "e9aee6181ae7199c2ce20f8355442e61"
server_key_version: "encryptionkey@0"
I20260925 07:21:01.886948 10063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:01.903258 10063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:01.905628 10063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:01.905784 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:01.906018 10063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:01.906315 10063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:01.906396 10063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.906437 10063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:01.906466 10063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:01.912531 10063 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:37897
I20260925 07:21:01.912578 10191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:37897 every 8 connection(s)
I20260925 07:21:01.912969 10063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:01.914350 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10063
I20260925 07:21:01.914431 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:01.914624 4284 external_mini_cluster.cc:1428] Setting key 09c7f553a037e346805943f8c3162f0c
Sep 25 07:21:01 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320861, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
I20260925 07:21:01.923977 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:01.913725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:44149 (local address 127.4.47.62:33207)
0925 07:21:01.913769 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:01.913771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:01.913941 (+ 170us) server_negotiation.cc:407] Connection header received
0925 07:21:01.915010 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:01.915012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:01.915024 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:01.915135 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:01.915806 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.916567 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.918070 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.918400 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.918474 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:01.921026 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:01.921027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:01.921028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:01.921034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:01.922488 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.922969 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.922970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.922970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.923019 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:01.923389 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:01.923390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:01.923390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:01.923493 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:01.923547 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:01.923849 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 07:21:01.923909 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:21:01.924165 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:01.913680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33207 (local address 127.4.47.2:44149)
0925 07:21:01.913831 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:01.913873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:01.914896 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:01.915205 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:01.915209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:01.915523 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:01.915709 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:01.915714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.916665 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.916666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:01.917981 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:01.917984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:01.918480 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:01.918481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:01.918574 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:01.919134 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:01.919150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:01.920935 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:01.922576 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:01.922583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:01.922586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:01.922893 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:01.923147 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:01.923148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:01.923149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:01.923304 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:01.923567 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:01.923573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:01.923717 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 07:21:01.923814 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 07:21:01.923932 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260925 07:21:01.924619 10192 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33207
I20260925 07:21:01.924702 10192 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:01.924894 10192 heartbeater.cc:511] Master 127.4.47.62:33207 requested a full tablet report, sending...
I20260925 07:21:01.925285 9867 ts_manager.cc:194] Registered new tserver with Master: 4754c33188e4441bbcd4428f8b82c36c (127.4.47.2:37897)
I20260925 07:21:01.925658 9867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:44149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:21:01.941589 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:02.022037 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:02.022257 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:02.022282 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:02.023522 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:02.023567 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:02.023595 10197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:02.025142 10197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:02.025413 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:02.025662 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:02.030812 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:02.030814 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:02.030812 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:02.030871 10197 server_base.cc:1034] running on GCE node
I20260925 07:21:02.031265 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:02.031464 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:02.032598 10197 hybrid_clock.cc:648] HybridClock initialized: now 1790320862032583 us; error 28 us; skew 500 ppm
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.035105 10197 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:21:02.035578 10197 webserver.cc:460] Webserver started at http://127.4.47.3:46681/ using document root <none> and password file <none>
I20260925 07:21:02.035822 10197 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:02.035892 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:02.036048 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:02.037110 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "0e3f8e1b6eb544689107bf97590d04de"
format_stamp: "Formatted at 2026-09-25 07:21:02 on dist-test-slave-zw07"
server_key: "069a8078d01a4f869beb91beca10abc8"
server_key_iv: "636b1555080a9b236bcf46b81d8378f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:02.037479 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "0e3f8e1b6eb544689107bf97590d04de"
format_stamp: "Formatted at 2026-09-25 07:21:02 on dist-test-slave-zw07"
server_key: "069a8078d01a4f869beb91beca10abc8"
server_key_iv: "636b1555080a9b236bcf46b81d8378f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:02.038829 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:02.039666 10212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:02.039851 10197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:02.039954 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "0e3f8e1b6eb544689107bf97590d04de"
format_stamp: "Formatted at 2026-09-25 07:21:02 on dist-test-slave-zw07"
server_key: "069a8078d01a4f869beb91beca10abc8"
server_key_iv: "636b1555080a9b236bcf46b81d8378f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:02.040045 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:02.052841 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:02.054924 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:02.055189 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:02.055454 10197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:02.055779 10197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:02.055833 10197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:02.055876 10197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:02.055907 10197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:02.061910 10197 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:40591
I20260925 07:21:02.061966 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:40591 every 8 connection(s)
I20260925 07:21:02.062309 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:02.066293 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10197
I20260925 07:21:02.066385 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:02.066565 4284 external_mini_cluster.cc:1428] Setting key 2cb0aa52fa3065acb1c1bb94e03a81e2
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:02.071794 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:02.063132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:56727 (local address 127.4.47.62:33207)
0925 07:21:02.063213 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.063216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.063498 (+ 282us) server_negotiation.cc:407] Connection header received
0925 07:21:02.064417 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.064418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.064429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.064484 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.065080 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.065574 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.066776 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.067034 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.067165 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:02.069352 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:02.069354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:02.069354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:02.069360 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:02.070532 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.070895 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.070896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.070897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.070961 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.071262 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.071262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.071263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.071366 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:02.071422 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:02.071654 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 07:21:02.071728 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:21:02.071954 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:02.063101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33207 (local address 127.4.47.3:56727)
0925 07:21:02.063345 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:02.063382 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:02.064289 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:02.064541 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:02.064545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:02.064809 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:02.065008 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:02.065012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.065673 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.065674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:02.066686 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:02.066689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.067175 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.067177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:02.067314 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:02.067877 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:02.067886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:02.069269 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:02.070600 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:02.070605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:02.070607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:02.070820 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:02.071019 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:02.071020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:02.071020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:02.071201 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:02.071424 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:02.071428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:02.071543 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 07:21:02.071625 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:21:02.071743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":109,"threads_started":1}
I20260925 07:21:02.072543 10326 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33207
I20260925 07:21:02.072655 10326 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:02.072830 10326 heartbeater.cc:511] Master 127.4.47.62:33207 requested a full tablet report, sending...
I20260925 07:21:02.073256 9867 ts_manager.cc:194] Registered new tserver with Master: 0e3f8e1b6eb544689107bf97590d04de (127.4.47.3:40591)
I20260925 07:21:02.073650 9867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:56727
I20260925 07:21:02.078503 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/test-admin.keytab.
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.094854 4284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 07:21:02.096553 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:02.095795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.4.47.1:33059)
0925 07:21:02.095978 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.095983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.095993 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:02.096017 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.096019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.096032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.096097 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.096445 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47252: BlockingRecv error: recv got EOF from 127.0.0.1:47252 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
W20260925 07:21:02.096638 4284 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc' of type FILE
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.097605 4284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.098479 4284 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:21:02.106443 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:02.098966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.4.47.1:33059)
0925 07:21:02.099022 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.099024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.099135 (+ 111us) server_negotiation.cc:407] Connection header received
0925 07:21:02.099182 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.099182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.099193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.099271 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.099662 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.100379 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.101114 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.101452 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.101545 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:02.103601 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:02.103610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:02.103610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:02.103617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:02.105228 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.105600 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.105604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.105605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.105667 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.105879 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.105879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.105880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.106021 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:02.106174 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:02.106253 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 07:21:02.106329 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 07:21:02.108265 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:02.107791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.4.47.1:33059)
0925 07:21:02.107830 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.107832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.107841 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:02.107901 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.107901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.107909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.107956 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.108198 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47270: BlockingRecv error: recv got EOF from 127.0.0.1:47270 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:21:02.108368 4284 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc' of type FILE
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.109350 4284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.110311 4284 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:21:02.117779 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:02.110942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.4.47.1:33059)
0925 07:21:02.111002 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.111003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.111019 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:21:02.111042 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.111043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.111057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.111176 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.111531 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.112030 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.112748 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.112948 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.113012 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:02.115393 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:02.115395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:02.115395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:02.115401 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:02.116721 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.117034 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.117035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.117035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.117085 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.117247 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.117248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.117248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.117383 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:02.117457 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:02.117561 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:21:02.117620 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 07:21:02.119374 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:02.118716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.4.47.1:33059)
0925 07:21:02.118749 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.118751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.118777 (+ 26us) server_negotiation.cc:407] Connection header received
0925 07:21:02.118806 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.118806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.118814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.118853 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.119221 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47286: BlockingRecv error: recv got EOF from 127.0.0.1:47286 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:21:02.119433 4284 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790320837143632-4284-0/krb5kdc/krb5cc)
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.120502 4284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:02.121474 4284 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[9822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:21:02.128484 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:02.122010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.4.47.1:33059)
0925 07:21:02.122061 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:02.122063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:02.122132 (+ 69us) server_negotiation.cc:407] Connection header received
0925 07:21:02.122208 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:02.122209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:02.122221 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:02.122289 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:02.122653 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.123182 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.123873 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:02.124064 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:02.124163 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:02.126120 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:02.126122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:02.126122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:02.126129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:02.127538 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.127838 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.127839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.127839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.127887 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:02.128048 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:02.128048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:02.128049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:02.128134 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:02.128204 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:02.128286 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:21:02.128338 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:21:02.129136 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9929
I20260925 07:21:02.134646 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10063
I20260925 07:21:02.139421 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10197
I20260925 07:21:02.143807 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 9835
2026-09-25T07:21:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1852 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[10346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[10346](info): set up 2 sockets
Sep 25 07:21:02 dist-test-slave-zw07 krb5kdc[10346](info): commencing operation
krb5kdc: starting...
W20260925 07:21:03.187808 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab.
Entry for principal oryx/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab.
I20260925 07:21:03.306865 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45079
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab
--principal=oryx/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:03.387902 10359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:03.388103 10359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:03.388125 10359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:03.389199 10359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:03.389240 10359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:03.389254 10359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:03.389267 10359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:03.389279 10359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:03.390668 10359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab
--principal=oryx/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:03.390852 10359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:03.391011 10359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:03.395812 10364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.395884 10367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.395805 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.396112 10359 server_base.cc:1034] running on GCE node
I20260925 07:21:03.396319 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:03.396555 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:03.397691 10359 hybrid_clock.cc:648] HybridClock initialized: now 1790320863397679 us; error 27 us; skew 500 ppm
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:03.400143 10359 init.cc:377] Logged in from keytab as oryx/127.4.47.62@KRBTEST.COM (short username oryx)
I20260925 07:21:03.400527 10359 webserver.cc:460] Webserver started at http://127.4.47.62:46595/ using document root <none> and password file <none>
I20260925 07:21:03.400738 10359 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:03.400779 10359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:03.400880 10359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:03.401906 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "f596c959984e45428d3656d6bde12096"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "d10cf158b0fc9dd512d1d5152ac8948d"
server_key_iv: "ec7fea64d87b2499840f8a4956efc4b6"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.402199 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "f596c959984e45428d3656d6bde12096"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "d10cf158b0fc9dd512d1d5152ac8948d"
server_key_iv: "ec7fea64d87b2499840f8a4956efc4b6"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.403491 10359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:21:03.404181 10374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.404354 10359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:03.404444 10359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "f596c959984e45428d3656d6bde12096"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "d10cf158b0fc9dd512d1d5152ac8948d"
server_key_iv: "ec7fea64d87b2499840f8a4956efc4b6"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.404500 10359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:03.412281 10359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:03.414091 10359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:03.414203 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:03.418154 10359 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:45079
I20260925 07:21:03.418231 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:45079 every 8 connection(s)
I20260925 07:21:03.418576 10359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:03.419648 10427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:03.421089 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10359
I20260925 07:21:03.421172 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:03.421370 4284 external_mini_cluster.cc:1428] Setting key fb26db729ad6b7ff38fbff3f00e2bea7
I20260925 07:21:03.422163 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Bootstrap starting.
I20260925 07:21:03.422734 10427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:03.423033 10427 log.cc:824] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:03.423748 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: No bootstrap required, opened a new log
I20260925 07:21:03.425518 10427 raft_consensus.cc:348] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } }
I20260925 07:21:03.425675 10427 raft_consensus.cc:374] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:03.425720 10427 raft_consensus.cc:729] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f596c959984e45428d3656d6bde12096, State: Initialized, Role: FOLLOWER
I20260925 07:21:03.425879 10427 consensus_queue.cc:260] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } }
I20260925 07:21:03.425985 10427 raft_consensus.cc:388] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:03.426025 10427 raft_consensus.cc:482] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:03.426412 10427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:03.427222 10427 raft_consensus.cc:504] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } }
I20260925 07:21:03.427330 10427 leader_election.cc:302] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f596c959984e45428d3656d6bde12096; no voters:
I20260925 07:21:03.427518 10427 leader_election.cc:288] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:03.427577 10432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:03.427723 10432 raft_consensus.cc:686] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 LEADER]: Becoming Leader. State: Replica: f596c959984e45428d3656d6bde12096, State: Running, Role: LEADER
I20260925 07:21:03.427783 10427 sys_catalog.cc:564] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:03.427814 10432 consensus_queue.cc:237] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } }
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:03.428151 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f596c959984e45428d3656d6bde12096. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } }
I20260925 07:21:03.428242 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:03.428169 10433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } }
I20260925 07:21:03.428495 10433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:03.429175 10441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:03.429672 10441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:03.431421 10441 catalog_manager.cc:1347] Generated new cluster ID: 5afab9184422469597fc0743cd344742
I20260925 07:21:03.431471 10441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:03.432333 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.421951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.4.47.62:45079)
0925 07:21:03.422134 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:03.422142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:03.422166 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:21:03.422639 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:03.422646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:03.422694 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:03.422853 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:03.423575 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.424531 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.425287 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.425719 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.425824 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.428767 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:03.428771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:03.428772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:03.428784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:03.430659 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.431094 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.431097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.431098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.431202 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.431448 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.431449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.431450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.431719 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:03.431894 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:03.432009 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 07:21:03.432154 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260925 07:21:03.460263 10441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:03.460815 10441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:03.467186 10441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Generated new TSK 0
I20260925 07:21:03.467386 10441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
I20260925 07:21:03.500170 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:03.581456 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:03.581689 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:03.581725 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:03.582849 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:03.582921 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:03.582964 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:03.584408 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:03.584676 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:03.584903 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:03.589787 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.589816 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.589789 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.590008 10453 server_base.cc:1034] running on GCE node
I20260925 07:21:03.590262 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:03.590521 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:03.591681 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790320863591672 us; error 45 us; skew 500 ppm
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:03.594082 10453 init.cc:377] Logged in from keytab as oryx/127.4.47.1@KRBTEST.COM (short username oryx)
I20260925 07:21:03.594581 10453 webserver.cc:460] Webserver started at http://127.4.47.1:37205/ using document root <none> and password file <none>
I20260925 07:21:03.594810 10453 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:03.594878 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:03.595029 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:03.596086 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "7a2c51a42550414894f20a023560b8e2"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "e899b4a0d7823428e3c3d87964a4e9b0"
server_key_iv: "563161220cd18abbd46a8c1c7b67cb6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.596447 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "7a2c51a42550414894f20a023560b8e2"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "e899b4a0d7823428e3c3d87964a4e9b0"
server_key_iv: "563161220cd18abbd46a8c1c7b67cb6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.597746 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:21:03.598441 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.598603 10453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:03.598699 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "7a2c51a42550414894f20a023560b8e2"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "e899b4a0d7823428e3c3d87964a4e9b0"
server_key_iv: "563161220cd18abbd46a8c1c7b67cb6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.598780 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:03.620980 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:03.622841 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:03.623015 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:03.623281 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:03.623606 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:03.623665 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.623706 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:03.623751 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.629716 10453 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:44947
I20260925 07:21:03.629765 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:44947 every 8 connection(s)
I20260925 07:21:03.630075 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:21:03.635448 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10453
I20260925 07:21:03.635608 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:03.635820 4284 external_mini_cluster.cc:1428] Setting key c2b39e8afda81e02c9e9f2534e8ec39a
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:03.639809 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.630737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:57751 (local address 127.4.47.62:45079)
0925 07:21:03.630801 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:03.630803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:03.631006 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:21:03.631505 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:03.631506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:03.631516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:03.631575 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:03.632474 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.632976 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.633950 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.634146 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.634197 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.636724 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:03.636726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:03.636726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:03.636735 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:03.638319 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.638829 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.638830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.638830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.638880 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.639183 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.639184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.639184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.639287 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:03.639338 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:03.639676 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 07:21:03.639744 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:21:03.640265 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.630709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:57751)
0925 07:21:03.630871 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:03.630913 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:03.631414 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:03.631628 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:03.631632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:03.632183 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:03.632390 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.632395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.633054 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.633055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.633866 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.633868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.634216 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.634217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.634315 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.634803 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:03.634812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:03.636635 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:03.638418 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.638425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.638427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.638752 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.638942 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.638943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.638943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.639115 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.639352 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:03.639358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:03.639532 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 07:21:03.639642 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 07:21:03.639823 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 07:21:03.640853 10582 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:03.640971 10582 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:03.641149 10582 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:03.641675 10391 ts_manager.cc:194] Registered new tserver with Master: 7a2c51a42550414894f20a023560b8e2 (127.4.47.1:44947)
I20260925 07:21:03.642285 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:57751
WARNING: no policy specified for oryx/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
I20260925 07:21:03.661790 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:03.741963 10587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:03.742175 10587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:03.742198 10587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:03.743294 10587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:03.743330 10587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:03.743356 10587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:03.744730 10587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:03.744915 10587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:03.745086 10587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:03.750065 10593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.750188 10587 server_base.cc:1034] running on GCE node
W20260925 07:21:03.750121 10595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.750085 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.750576 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:03.750836 10587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:03.751991 10587 hybrid_clock.cc:648] HybridClock initialized: now 1790320863751962 us; error 29 us; skew 500 ppm
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:03.754392 10587 init.cc:377] Logged in from keytab as oryx/127.4.47.2@KRBTEST.COM (short username oryx)
I20260925 07:21:03.754834 10587 webserver.cc:460] Webserver started at http://127.4.47.2:37269/ using document root <none> and password file <none>
I20260925 07:21:03.755142 10587 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:03.755221 10587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:03.755358 10587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:03.756433 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "35f972d202cc4affb758f28baf1b014d"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "a3a5469375162837243689243447a253"
server_key_iv: "12a45401e8b31371a1c6b026e2f0afbf"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.756834 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "35f972d202cc4affb758f28baf1b014d"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "a3a5469375162837243689243447a253"
server_key_iv: "12a45401e8b31371a1c6b026e2f0afbf"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.758145 10587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:03.758865 10602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.759037 10587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:03.759169 10587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "35f972d202cc4affb758f28baf1b014d"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "a3a5469375162837243689243447a253"
server_key_iv: "12a45401e8b31371a1c6b026e2f0afbf"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.759260 10587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:03.778105 10587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:03.780085 10587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:03.780251 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:03.780493 10587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:03.780850 10587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:03.780886 10587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.780938 10587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:03.780964 10587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.786715 10587 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:34957
I20260925 07:21:03.786778 10715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:34957 every 8 connection(s)
I20260925 07:21:03.787132 10587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:03.796447 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.787982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55043 (local address 127.4.47.62:45079)
0925 07:21:03.788059 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:03.788061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:03.788299 (+ 238us) server_negotiation.cc:407] Connection header received
0925 07:21:03.788759 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:03.788760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:03.788770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:03.788820 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:03.789752 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.790253 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.791328 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.791538 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.791595 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.793749 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:03.793751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:03.793751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:03.793759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:03.794994 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.795423 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.795424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.795425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.795488 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.795748 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.795749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.795749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.795871 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:03.795951 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:03.796232 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 07:21:03.796317 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:21:03.796594 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.787943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:55043)
0925 07:21:03.788141 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:03.788194 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:03.788674 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:03.788882 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:03.788885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:03.789477 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:03.789668 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.789673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.790337 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.790338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.791236 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.791239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.791611 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.791612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.791714 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.792163 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:03.792171 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:03.793646 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:03.795095 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.795102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.795105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.795336 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.795562 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.795563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.795563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.795671 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.795942 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:03.795948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:03.796087 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 07:21:03.796193 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 07:21:03.796354 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260925 07:21:03.796672 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10587
I20260925 07:21:03.796782 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:03.797053 4284 external_mini_cluster.cc:1428] Setting key 898f6cb95f3c021d0e1ca30e1e6d8879
I20260925 07:21:03.797122 10716 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:03.797226 10716 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:03.797457 10716 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:03.797864 10391 ts_manager.cc:194] Registered new tserver with Master: 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957)
I20260925 07:21:03.798246 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:55043
WARNING: no policy specified for oryx/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
I20260925 07:21:03.822250 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:03.903863 10721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:03.904084 10721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:03.904109 10721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:03.905283 10721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:03.905331 10721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:03.905360 10721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:03.906972 10721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:03.907366 10721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:03.907657 10721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:03.912837 10726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:03.912885 10729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.913025 10721 server_base.cc:1034] running on GCE node
W20260925 07:21:03.912873 10727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:03.913333 10721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:03.913578 10721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:03.914764 10721 hybrid_clock.cc:648] HybridClock initialized: now 1790320863914742 us; error 36 us; skew 500 ppm
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:03.917402 10721 init.cc:377] Logged in from keytab as oryx/127.4.47.3@KRBTEST.COM (short username oryx)
I20260925 07:21:03.917863 10721 webserver.cc:460] Webserver started at http://127.4.47.3:35979/ using document root <none> and password file <none>
I20260925 07:21:03.918084 10721 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:03.918150 10721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:03.918303 10721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:03.919425 10721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "e124957559204d0ea240a539dd2d1c4f"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "6ccf019378d5da4920c4f20f8fd59320"
server_key_iv: "e6344c72eb15aabd934fec4cd7626dd0"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.919776 10721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "e124957559204d0ea240a539dd2d1c4f"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "6ccf019378d5da4920c4f20f8fd59320"
server_key_iv: "e6344c72eb15aabd934fec4cd7626dd0"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.921092 10721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:21:03.921900 10736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.922068 10721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:03.922168 10721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "e124957559204d0ea240a539dd2d1c4f"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "6ccf019378d5da4920c4f20f8fd59320"
server_key_iv: "e6344c72eb15aabd934fec4cd7626dd0"
server_key_version: "encryptionkey@0"
I20260925 07:21:03.922238 10721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:03.934062 10721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:03.936499 10721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:03.936668 10721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:03.936910 10721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:03.937222 10721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:03.937278 10721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.937321 10721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:03.937350 10721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:03.943039 10721 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:46745
I20260925 07:21:03.943106 10849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:46745 every 8 connection(s)
I20260925 07:21:03.943488 10721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:03.947346 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10721
I20260925 07:21:03.947445 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:03.947616 4284 external_mini_cluster.cc:1428] Setting key 46e52bb952fff0630aeed825a5ffb90a
Sep 25 07:21:03 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:03.952934 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.944230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:37101 (local address 127.4.47.62:45079)
0925 07:21:03.944294 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:03.944296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:03.944499 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:21:03.945007 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:03.945008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:03.945020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:03.945082 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:03.945985 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.946507 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.947751 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.948000 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.948086 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.950162 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:03.950163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:03.950163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:03.950169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:03.951726 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.952100 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.952101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.952102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.952158 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:03.952401 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:03.952402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:03.952402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:03.952519 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:03.952579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:03.952804 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 07:21:03.952870 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:03.953096 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:03.944193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:37101)
0925 07:21:03.944363 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:03.944405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:03.944869 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:03.945139 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:03.945143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:03.945710 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:03.945906 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.945910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.946577 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.946578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.947672 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:03.947675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:03.948113 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:03.948114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:03.948266 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:03.948758 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:03.948766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:03.950062 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:03.951797 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.951802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.951804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.952013 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.952228 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:03.952228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:03.952229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:03.952324 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:03.952579 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:03.952583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:03.952695 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 07:21:03.952775 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 07:21:03.952893 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
I20260925 07:21:03.953576 10850 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:03.953672 10850 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:03.953812 10850 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:03.954185 10391 ts_manager.cc:194] Registered new tserver with Master: e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:03.954605 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:37101
I20260925 07:21:03.959453 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.47.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.61.keytab.
Entry for principal oryx/127.4.47.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.61.keytab.
I20260925 07:21:03.984145 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:40589
--webserver_interface=127.4.47.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.61.keytab
--principal=oryx/127.4.47.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45079,127.4.47.61:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:04.067415 10855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:04.067662 10855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:04.067696 10855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:04.068828 10855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:04.068902 10855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:04.068925 10855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:04.068945 10855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:04.068974 10855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:04.070447 10855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45079,127.4.47.61:40589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.61.keytab
--principal=oryx/127.4.47.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.47.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
W20260925 07:21:04.070590 10855 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.47.62:45079,127.4.47.61:40589'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 07:21:04.070761 10855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:04.070989 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.075821 10860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.076001 10855 server_base.cc:1034] running on GCE node
W20260925 07:21:04.075847 10861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.075827 10863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.076323 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.076596 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.077734 10855 hybrid_clock.cc:648] HybridClock initialized: now 1790320864077725 us; error 30 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.080305 10855 init.cc:377] Logged in from keytab as oryx/127.4.47.61@KRBTEST.COM (short username oryx)
I20260925 07:21:04.080772 10855 webserver.cc:460] Webserver started at http://127.4.47.61:33447/ using document root <none> and password file <none>
I20260925 07:21:04.080993 10855 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.081058 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.081218 10855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:04.082322 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/instance:
uuid: "1e009b31a67c41ad85a684300be2a398"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "2438f134657139b3e648553b8383830a"
server_key_iv: "9df7edd0f2c3cdd456784dfe64d62a89"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.082690 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal/instance:
uuid: "1e009b31a67c41ad85a684300be2a398"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "2438f134657139b3e648553b8383830a"
server_key_iv: "9df7edd0f2c3cdd456784dfe64d62a89"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.084108 10855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:04.084825 10870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.084988 10855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.085086 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "1e009b31a67c41ad85a684300be2a398"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "2438f134657139b3e648553b8383830a"
server_key_iv: "9df7edd0f2c3cdd456784dfe64d62a89"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.085156 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.111496 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.113976 10855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:04.114145 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.118417 10855 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.61:40589
I20260925 07:21:04.118491 10922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.61:40589 every 8 connection(s)
I20260925 07:21:04.118842 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
I20260925 07:21:04.119240 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 10855
I20260925 07:21:04.119944 10923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:04.121759 10923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.61@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:04.130328 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.119889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.4.47.61:40589)
0925 07:21:04.120081 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.120091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.120129 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:21:04.120559 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.120566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.120613 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.120777 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.121388 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.122462 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.123397 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.123858 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.123937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.126777 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.126783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.126784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.126789 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.128629 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129027 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.129030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.129030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.129106 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129323 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.129324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.129325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.129674 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.129793 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.129950 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.130069 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260925 07:21:04.130596 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.122334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42426)
0925 07:21:04.122588 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.122625 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.122676 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.122923 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.122925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.123249 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.123396 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.123397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.124105 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.124106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.124684 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.124686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.125209 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.125210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.125371 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.125854 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.125862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.127371 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.129317 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.129325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.129611 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.129840 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.129841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.129910 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.130232 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.130237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.130297 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.130385 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.130494 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
I20260925 07:21:04.130949 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.122433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.4.47.62:45079)
0925 07:21:04.122510 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.122512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.122699 (+ 187us) server_negotiation.cc:407] Connection header received
0925 07:21:04.122732 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.122733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.122746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.122819 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.123484 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.123999 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.124775 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.125107 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.125189 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.127458 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.127460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.127460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.127468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.129180 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129684 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.129685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.129685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.129760 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.129981 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.129982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.129983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.130174 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.130228 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.130430 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.130518 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:21:04.131477 10923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.61@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
I20260925 07:21:04.137580 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.131729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:47376)
0925 07:21:04.131778 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.131788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.131825 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.131928 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.131928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.132062 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.132101 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.132102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.132737 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.132739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.133215 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.133216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.133591 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.133592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.133653 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.134023 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.134027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.135262 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.136559 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.136559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.136560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.136723 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.136960 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.136961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.136961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.137012 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.137271 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.137271 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.137308 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.137375 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.137470 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:21:04.137626 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.131800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.4.47.61:40589)
0925 07:21:04.131834 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.131835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.131846 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:21:04.131865 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.131866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.131874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.131925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.132174 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.132652 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.133306 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.133510 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.133591 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.135367 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.135368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.135368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.135375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.136465 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.136831 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.136832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.136833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.136887 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.137079 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.137080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.137080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.137188 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.137276 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.137399 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.137519 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":19}
I20260925 07:21:04.138955 10923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Bootstrap starting.
I20260925 07:21:04.139812 10923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:04.140030 10923 log.cc:824] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:04.140692 10923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: No bootstrap required, opened a new log
I20260925 07:21:04.142076 10923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } }
I20260925 07:21:04.142198 10923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:04.142223 10923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e009b31a67c41ad85a684300be2a398, State: Initialized, Role: FOLLOWER
I20260925 07:21:04.142350 10923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } }
I20260925 07:21:04.142599 10923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:04.142596 10929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } }
I20260925 07:21:04.142743 10929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:21:04.144281 10940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:21:04.144373 10940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:21:04.145886 10855 master_runner.cc:428] Detected that this master 1e009b31a67c41ad85a684300be2a398 is joining an existing cluster
I20260925 07:21:04.145938 10855 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.47.61:40589
I20260925 07:21:04.153591 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.146935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:47382)
0925 07:21:04.147381 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.147389 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.147445 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.147592 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.147593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.147801 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.147879 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.147881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.148567 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.148568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.149284 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.149286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.149747 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.149748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.149825 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.150288 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.150293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.150703 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.152538 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.152539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.152540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.152719 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.152930 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.152931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.152931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.153000 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.153229 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.153230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.153277 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.153360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.153460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":2,"spinlock_wait_cycles":8960,"thread_start_us":92,"threads_started":1}
I20260925 07:21:04.153591 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.147129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.4.47.61:40589)
0925 07:21:04.147171 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.147173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.147428 (+ 255us) server_negotiation.cc:407] Connection header received
0925 07:21:04.147501 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.147501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.147510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.147574 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.147958 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.148488 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.149372 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.149664 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.149735 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.150793 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.150794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.150795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.150801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.152443 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.152809 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.152810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.152810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.152865 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.153060 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.153061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.153062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.153164 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.153238 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.153392 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.153509 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":19}
I20260925 07:21:04.154271 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.147009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.4.47.62:45079)
0925 07:21:04.147057 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.147059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.147287 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:21:04.147344 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.147344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.147354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.147427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.148143 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.148902 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.149812 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.150099 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.150166 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.151294 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.151296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.151296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.151303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.152838 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.153262 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.153264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.153264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.153320 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.153519 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.153520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.153520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.153639 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.153692 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.154003 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.154085 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:21:04.154603 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.146936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42430)
0925 07:21:04.147204 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.147214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.147284 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.147432 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.147432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.147990 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.148054 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.148055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.148988 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.148989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.149688 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.149690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.150217 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.150218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.150307 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.150716 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.150721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.151179 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.152931 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.152932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.152932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.153162 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.153397 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.153397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.153398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.153452 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.153744 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.153744 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.153782 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.154396 (+ 614us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.154501 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.62@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
I20260925 07:21:04.164552 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.157106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.4.47.61:40589)
0925 07:21:04.157148 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.157150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.157328 (+ 178us) server_negotiation.cc:407] Connection header received
0925 07:21:04.157418 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.157419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.157427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.157473 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.157950 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.158369 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.159548 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.159717 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.159767 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.161836 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.161837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.161838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.161844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.163323 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.163719 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.163720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.163721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.163777 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.163982 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.163982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.163983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.164085 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.164154 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.164362 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.164445 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:21:04.164556 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.157056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:47390)
0925 07:21:04.157207 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.157260 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.157336 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.157478 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.157481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.157817 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.157877 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.157879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.158442 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.158443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.159474 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.159477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.159786 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.159787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.159865 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.160255 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.160260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.161739 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.163420 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.163427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.163429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.163630 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.163843 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.163844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.163844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.163912 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.164159 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.164166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.164237 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.164337 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.164448 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260925 07:21:04.165225 10391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.47.61:40589: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1e009b31a67c41ad85a684300be2a398"
member_type: NON_VOTER
last_known_addr {
host: "127.4.47.61"
port: 40589
}
attrs {
promote: true
}
}
dest_uuid: "f596c959984e45428d3656d6bde12096"
cas_config_opid_index: -1
I20260925 07:21:04.165544 10391 consensus_queue.cc:237] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } }
I20260925 07:21:04.166038 10950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e009b31a67c41ad85a684300be2a398 (127.4.47.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.166285 10897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:04.166977 10950 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.47.61:40589
I20260925 07:21:04.167205 10855 master.cc:537] Master@127.4.47.61:40589 shutting down...
I20260925 07:21:04.167229 10897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 07:21:04.167259 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.168326 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:04.167557 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260925 07:21:04.169490 10375 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 07:21:04.169937 10855 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 07:21:04.169962 10375 consensus_peers.cc:487] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 -> Peer 1e009b31a67c41ad85a684300be2a398 (127.4.47.61:40589): Couldn't send request to peer 1e009b31a67c41ad85a684300be2a398. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:21:04.170035 10855 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:04.170089 10855 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: stopping tablet replica
I20260925 07:21:04.172629 10855 master.cc:552] Master@127.4.47.61:40589 shutdown complete.
I20260925 07:21:04.172677 10855 master_runner.cc:305] Clearing existing system tablet
I20260925 07:21:04.172813 10855 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.172860 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.173532 10855 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.174036 10954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.174168 10855 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.174226 10855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.174252 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "1e009b31a67c41ad85a684300be2a398"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "2438f134657139b3e648553b8383830a"
server_key_iv: "9df7edd0f2c3cdd456784dfe64d62a89"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.174945 10855 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:04.176571 10855 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 07:21:04.176640 10855 log.cc:1196] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 07:21:04.176834 10855 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Deleting consensus metadata
I20260925 07:21:04.176962 10855 master_runner.cc:315] Copying system tablet from 127.4.47.62:45079
I20260925 07:21:04.177541 10855 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: tablet copy: Beginning tablet copy session from remote peer at address 127.4.47.62:45079
I20260925 07:21:04.183863 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.177890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.4.47.62:45079)
0925 07:21:04.177937 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.177939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.177988 (+ 49us) server_negotiation.cc:407] Connection header received
0925 07:21:04.178062 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.178063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.178071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.178118 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.178439 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.178918 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.179963 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.180137 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.180188 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.181206 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.181219 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.181220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.181228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.182770 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.183172 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.183173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.183174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.183231 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.183422 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.183423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.183423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.183520 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.183569 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.183743 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.183802 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:21:04.183928 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.177794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42432)
0925 07:21:04.177926 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.177938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.178003 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.178123 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.178124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.178314 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.178370 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.178372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.178996 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.178997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.179886 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.179888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.180206 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.180207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.180278 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.180687 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.180692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.181079 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.182850 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.182851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.182851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.183098 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.183291 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.183292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.183293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.183356 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.183580 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.183581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.183626 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.183717 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.183826 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":91,"threads_started":1}
I20260925 07:21:04.184235 10411 tablet_copy_service.cc:140] P f596c959984e45428d3656d6bde12096: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1e009b31a67c41ad85a684300be2a398 ({username='oryx', principal='oryx/127.4.47.61@KRBTEST.COM'} at 127.0.0.1:42432)
I20260925 07:21:04.184334 10411 tablet_copy_service.cc:161] P f596c959984e45428d3656d6bde12096: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1e009b31a67c41ad85a684300be2a398 at {username='oryx', principal='oryx/127.4.47.61@KRBTEST.COM'} at 127.0.0.1:42432: session id = 1e009b31a67c41ad85a684300be2a398-00000000000000000000000000000000
I20260925 07:21:04.185129 10411 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:21:04.185724 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 07:21:04.186638 10855 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: tablet copy: Starting download of 0 data blocks...
I20260925 07:21:04.186789 10855 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: tablet copy: Starting download of 1 WAL segments...
I20260925 07:21:04.187604 10855 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:21:04.188305 10411 tablet_copy_service.cc:342] P f596c959984e45428d3656d6bde12096: Request end of tablet copy session 1e009b31a67c41ad85a684300be2a398-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.47.61@KRBTEST.COM'} at 127.0.0.1:42432
I20260925 07:21:04.188360 10411 tablet_copy_service.cc:433] P f596c959984e45428d3656d6bde12096: ending tablet copy session 1e009b31a67c41ad85a684300be2a398-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1e009b31a67c41ad85a684300be2a398
I20260925 07:21:04.190606 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.191774 10962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.191781 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.191972 10961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.195237 10855 server_base.cc:1034] running on GCE node
I20260925 07:21:04.195363 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.195504 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.196635 10855 hybrid_clock.cc:648] HybridClock initialized: now 1790320864196626 us; error 30 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.198073 10855 init.cc:377] Logged in from keytab as oryx/127.4.47.61@KRBTEST.COM (short username oryx)
I20260925 07:21:04.198388 10855 webserver.cc:460] Webserver started at http://127.4.47.61:38135/ using document root <none> and password file <none>
I20260925 07:21:04.198511 10855 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.198558 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.199218 10855 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.199759 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.199951 10855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.200022 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "1e009b31a67c41ad85a684300be2a398"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "2438f134657139b3e648553b8383830a"
server_key_iv: "9df7edd0f2c3cdd456784dfe64d62a89"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.200085 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.220233 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.221841 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.225086 10855 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.61:40589
I20260925 07:21:04.225203 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.61:40589 every 8 connection(s)
I20260925 07:21:04.225515 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
I20260925 07:21:04.226202 11024 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:21:04.226605 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Bootstrap starting.
I20260925 07:21:04.229130 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:21:04.229441 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: Bootstrap complete.
I20260925 07:21:04.229646 11024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } }
I20260925 07:21:04.229737 11024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e009b31a67c41ad85a684300be2a398, State: Initialized, Role: LEARNER
I20260925 07:21:04.229789 11024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } }
I20260925 07:21:04.229990 11024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:04.229974 11027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.230070 11027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:21:04.231441 11038 catalog_manager.cc:1259] Loaded cluster ID: 5afab9184422469597fc0743cd344742
I20260925 07:21:04.231510 11038 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: loading cluster ID for follower catalog manager: success
I20260925 07:21:04.232944 11038 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: acquiring CA information for follower catalog manager: success
I20260925 07:21:04.233276 11038 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:21:04.239451 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.232255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.4.47.61:40589)
0925 07:21:04.232385 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.232387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.232421 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:21:04.232493 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.232493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.232506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.232626 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.233318 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.234296 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.235006 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.235333 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.235405 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.236659 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.236661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.236661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.236668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.238289 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.238618 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.238619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.238620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.238685 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.238879 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.238880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.238880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.239012 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.239140 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.239254 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.239350 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.4.47.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.60.keytab.
Entry for principal oryx/127.4.47.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.60.keytab.
I20260925 07:21:04.263833 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:41083
--webserver_interface=127.4.47.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.60.keytab
--principal=oryx/127.4.47.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:04.349402 11042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:04.349609 11042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:04.349632 11042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:04.350801 11042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:04.350844 11042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:04.350858 11042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:04.350870 11042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:04.350883 11042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:04.352415 11042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.60.keytab
--principal=oryx/127.4.47.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.47.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:04.352680 11042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:04.352922 11042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.357859 11048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.357945 11042 server_base.cc:1034] running on GCE node
W20260925 07:21:04.357867 11047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.357890 11050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.358383 11042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.358655 11042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.359792 11042 hybrid_clock.cc:648] HybridClock initialized: now 1790320864359767 us; error 44 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.362394 11042 init.cc:377] Logged in from keytab as oryx/127.4.47.60@KRBTEST.COM (short username oryx)
I20260925 07:21:04.362927 11042 webserver.cc:460] Webserver started at http://127.4.47.60:43445/ using document root <none> and password file <none>
I20260925 07:21:04.363240 11042 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.363310 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.363454 11042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:04.364549 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/instance:
uuid: "09572b9180d146049f26d59a7dd4ae2a"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "3c5c61ce3938b51e78b76fdc3d75170c"
server_key_iv: "b95d1273d7e4b76d0469c250cf45b652"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.364943 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal/instance:
uuid: "09572b9180d146049f26d59a7dd4ae2a"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "3c5c61ce3938b51e78b76fdc3d75170c"
server_key_iv: "b95d1273d7e4b76d0469c250cf45b652"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.366290 11042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:04.366994 11057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.367177 11042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.367281 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "09572b9180d146049f26d59a7dd4ae2a"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "3c5c61ce3938b51e78b76fdc3d75170c"
server_key_iv: "b95d1273d7e4b76d0469c250cf45b652"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.367357 11042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.388334 11042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.390754 11042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:04.390924 11042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.394996 11042 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.60:41083
I20260925 07:21:04.395048 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.60:41083 every 8 connection(s)
I20260925 07:21:04.395440 11042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
I20260925 07:21:04.396557 11110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:04.398092 11110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:21:04.398834 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11042
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.60@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
I20260925 07:21:04.408334 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.398916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.4.47.62:45079)
0925 07:21:04.398966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.398969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.399034 (+ 65us) server_negotiation.cc:407] Connection header received
0925 07:21:04.399711 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.399712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.399724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.399799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.400643 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.401539 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.402528 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.402777 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.402828 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.405183 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.405184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.405185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.405193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.406715 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.407261 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.407262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.407263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.407319 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.407609 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.407610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.407610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.407747 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.407811 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.408150 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.408226 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:21:04.408568 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.398743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41218)
0925 07:21:04.398912 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.398954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.399596 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.399865 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.399867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.400261 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.400535 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.400542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.401647 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.401648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.402453 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.402455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.402846 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.402847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.402964 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.403521 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.403529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.405090 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.406808 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.406815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.406817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.407175 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.407407 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.407408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.407409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.407535 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.407813 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.407818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.407993 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.408111 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.408277 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 07:21:04.408996 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.399353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.4.47.60:41083)
0925 07:21:04.399508 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.399514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.399566 (+ 52us) server_negotiation.cc:407] Connection header received
0925 07:21:04.399670 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.399678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.399704 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.399796 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.400662 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.401876 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.402636 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.402977 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.403044 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.405816 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.405823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.405823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.405830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.407625 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.408005 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.408007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.408008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.408054 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.408335 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.408336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.408336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.408534 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.408709 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.408807 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.408884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260925 07:21:04.409308 11110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.60@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
I20260925 07:21:04.415469 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.409552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:60320)
0925 07:21:04.409591 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.409600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.409637 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.409822 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.409823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.409958 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.409999 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.410000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.410659 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.410660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.411245 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.411247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.411635 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.411636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.411699 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.412034 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.412038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.413151 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.414439 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.414440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.414441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.414605 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.414816 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.414817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.414817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.414872 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.415144 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.415145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.415185 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.415272 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.415373 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:21:04.415469 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.409632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.4.47.61:40589)
0925 07:21:04.409702 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.409704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.409712 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:04.409735 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.409736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.409745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.409820 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.410076 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.410579 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.411324 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.411552 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.411636 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.413245 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.413247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.413247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.413254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.414348 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.414688 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.414689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.414689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.414750 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.414943 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.414944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.414944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.415056 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.415153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.415300 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.415381 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:21:04.415992 11110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.60@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
I20260925 07:21:04.423341 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.416234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.0.0.1:46938)
0925 07:21:04.416284 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.416295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.416337 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.416438 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.416439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.416562 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.416586 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.416587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.417200 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.417201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.417665 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.417666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.419142 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.419144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.419205 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.419552 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.419557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.420829 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.422368 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.422369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.422369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.422514 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.422760 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.422761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.422761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.422806 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.423048 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.423048 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.423141 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.423203 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.423277 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:21:04.423431 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.416314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.4.47.60:41083)
0925 07:21:04.416345 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.416347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.416362 (+ 15us) server_negotiation.cc:407] Connection header received
0925 07:21:04.416381 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.416382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.416391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.416438 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.416655 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.417121 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.417750 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.417957 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.418059 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.420922 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.420923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.420924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.420930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.422294 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.422622 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.422623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.422623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.422683 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.422874 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.422874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.422875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.422992 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.423050 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.423229 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.423334 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":17}
I20260925 07:21:04.424778 11110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Bootstrap starting.
I20260925 07:21:04.425625 11110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:04.425911 11110 log.cc:824] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:04.426621 11110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: No bootstrap required, opened a new log
I20260925 07:21:04.428262 11110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } }
I20260925 07:21:04.428392 11110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:04.428413 11110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09572b9180d146049f26d59a7dd4ae2a, State: Initialized, Role: FOLLOWER
I20260925 07:21:04.428548 11110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } }
I20260925 07:21:04.428869 11110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:04.428828 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } } }
I20260925 07:21:04.428941 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:21:04.430670 11126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:21:04.430727 11126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:21:04.433032 11042 master_runner.cc:428] Detected that this master 09572b9180d146049f26d59a7dd4ae2a is joining an existing cluster
I20260925 07:21:04.433084 11042 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.47.60:41083
I20260925 07:21:04.441315 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.4.47.62:45079)
0925 07:21:04.434176 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.434179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.434265 (+ 86us) server_negotiation.cc:407] Connection header received
0925 07:21:04.434329 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.434330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.434342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.434406 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.434825 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.435606 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.436671 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.436984 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.437024 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.438278 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.438279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.438280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.438287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.440021 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.440443 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.440444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.440444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.440507 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.440733 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.440734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.440735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.440877 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.440955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.441127 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.441202 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:21:04.441332 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41222)
0925 07:21:04.434195 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.434207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.434262 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.434409 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.434410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.434630 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.434716 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.434718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.435688 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.435688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.436579 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.436580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.437111 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.437112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.437168 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.437670 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.437675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.438163 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.440103 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.440104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.440104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.440334 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.440585 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.440586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.440586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.440653 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.440947 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.440948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.440994 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.441094 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.441201 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260925 07:21:04.443198 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.4.47.61:40589)
0925 07:21:04.434753 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.434755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.434967 (+ 212us) server_negotiation.cc:407] Connection header received
0925 07:21:04.434991 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.434992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.435001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.435054 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.435511 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.436237 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.437234 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.437875 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.437923 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.439979 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.439981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.439981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.439988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.441924 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.442309 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.442310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.442310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.442368 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.442589 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.442589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.442590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.442716 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.442787 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.442970 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.443049 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:21:04.443651 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:60326)
0925 07:21:04.434869 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.434877 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.434934 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.435059 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.435059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.435360 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.435421 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.435423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.436327 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.436328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.437143 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.437145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.438003 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.438003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.438063 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.438727 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.438731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.439868 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.442011 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.442012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.442013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.442231 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.442445 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.442446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.442446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.442514 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.442789 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.442790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.442842 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.442923 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.443539 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":78,"threads_started":1}
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320863, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.62@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
I20260925 07:21:04.458871 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.444315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.0.0.1:46956)
0925 07:21:04.444353 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.444363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.444400 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.444641 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.444642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.444807 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.444863 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.444865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.445487 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.445488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.446335 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.446337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.446679 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.446680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.446754 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.447225 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.447230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.448514 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.457702 (+ 9188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.457703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.457704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.457948 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.458159 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.458160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.458226 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.458492 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.458493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.458544 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.458637 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.458758 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 07:21:04.459017 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.4.47.60:41083)
0925 07:21:04.434741 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.434743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.434749 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:21:04.434766 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.434767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.434777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.434831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.436071 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.436742 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.437781 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.438061 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.438102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.456112 (+ 18010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.456114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.456114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.456122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.457816 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458200 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.458201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.458202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.458243 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458416 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.458417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.458417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.458547 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.458615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.458764 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.458898 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":71}
I20260925 07:21:04.459178 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.434270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.0.0.1:46946)
0925 07:21:04.434475 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.434485 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.434525 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.435710 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.435710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.435925 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.435984 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.435986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.436833 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.436834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.437688 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.437690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.438168 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.438169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.438222 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.455568 (+ 17346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.455574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.456003 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.457909 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.457910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.457910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.458139 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.458305 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.458306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.458350 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.458619 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.458620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.458658 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.458726 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.458831 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260925 07:21:04.459303 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.444365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.4.47.60:41083)
0925 07:21:04.444519 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.444521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.444527 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:21:04.444555 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.444556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.444564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.444611 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.444943 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.445410 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.446415 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.446609 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.446657 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.448603 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.448604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.448604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.448612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.457600 (+ 8988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458024 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.458024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.458025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.458084 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.458280 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.458280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.458281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.458437 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.458496 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.459113 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.459181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 07:21:04.460016 10391 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.47.60:41083: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a"
member_type: NON_VOTER
last_known_addr {
host: "127.4.47.60"
port: 41083
}
attrs {
promote: true
}
}
dest_uuid: "f596c959984e45428d3656d6bde12096"
cas_config_opid_index: 5
I20260925 07:21:04.460198 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10453
I20260925 07:21:04.460273 10391 consensus_queue.cc:237] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
I20260925 07:21:04.460819 10950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.461905 10950 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.47.60:41083
I20260925 07:21:04.461984 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.462085 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:04.463415 11084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:04.464329 11084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 07:21:04.464977 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:04.465540 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.460953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:60342)
0925 07:21:04.461275 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.461285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.461329 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.461533 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.461534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.461535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:04.461580 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.461581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.462512 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.462513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.463907 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.463910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.464921 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.464922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.465309 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.465333 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.465388 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.465444 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260925 07:21:04.465797 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.461043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.4.47.61:40589)
0925 07:21:04.461245 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.461247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.461380 (+ 133us) server_negotiation.cc:407] Connection header received
0925 07:21:04.461405 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.461406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.461428 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.461483 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:04.461643 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.462426 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.464011 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.464731 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.464803 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.464884 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.465676 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.465715 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260925 07:21:04.465935 11042 master.cc:537] Master@127.4.47.60:41083 shutting down...
I20260925 07:21:04.465983 10998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 LEARNER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 07:21:04.466164 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 07:21:04.466279 11084 raft_consensus.cc:858] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
W20260925 07:21:04.466372 11084 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
I20260925 07:21:04.466706 10998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.466845 11084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e009b31a67c41ad85a684300be2a398 (127.4.47.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.467950 11027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f596c959984e45428d3656d6bde12096. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.468056 11027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:21:04.468096 11027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.468166 11027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:21:04.468660 10949 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: attempting to promote NON_VOTER 1e009b31a67c41ad85a684300be2a398 to VOTER
I20260925 07:21:04.468672 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f596c959984e45428d3656d6bde12096. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.468744 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:21:04.468781 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.468823 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:21:04.468855 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: NON_VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: true } } }
I20260925 07:21:04.468856 10949 consensus_queue.cc:237] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
I20260925 07:21:04.468914 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:21:04.469246 10998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 LEARNER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 07:21:04.469421 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 07:21:04.469769 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:44947
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:45079
--webserver_port=37205
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:21:04.470042 10950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e009b31a67c41ad85a684300be2a398 (127.4.47.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
W20260925 07:21:04.470373 10376 consensus_peers.cc:487] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 -> Peer 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60:41083): Couldn't send request to peer 09572b9180d146049f26d59a7dd4ae2a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:21:04.471156 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.471251 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:04.471460 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.471542 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:04.471766 10998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e009b31a67c41ad85a684300be2a398 (127.4.47.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.472594 11137 mvcc.cc:205] Tried to move back new op lower bound from 7333154256759775232 to 7333154256600756224. Current Snapshot: MvccSnapshot[applied={T|T < 7333154256611667968}]
I20260925 07:21:04.472687 11142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.472779 11142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:21:04.473600 11042 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 07:21:04.473711 11042 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 07:21:04.473753 11042 tablet_replica.cc:330] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: stopping tablet replica
I20260925 07:21:04.486702 11042 master.cc:552] Master@127.4.47.60:41083 shutdown complete.
I20260925 07:21:04.486776 11042 master_runner.cc:305] Clearing existing system tablet
I20260925 07:21:04.486913 11042 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.486974 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.487742 11042 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.488255 11149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.488372 11042 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.488421 11042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.488451 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "09572b9180d146049f26d59a7dd4ae2a"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "3c5c61ce3938b51e78b76fdc3d75170c"
server_key_iv: "b95d1273d7e4b76d0469c250cf45b652"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.489081 11042 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:04.490535 11042 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 07:21:04.490620 11042 log.cc:1196] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 07:21:04.490839 11042 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Deleting consensus metadata
I20260925 07:21:04.490995 11042 master_runner.cc:315] Copying system tablet from 127.4.47.62:45079
I20260925 07:21:04.491643 11042 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: tablet copy: Beginning tablet copy session from remote peer at address 127.4.47.62:45079
I20260925 07:21:04.497812 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.492032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.4.47.62:45079)
0925 07:21:04.492081 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.492083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.492154 (+ 71us) server_negotiation.cc:407] Connection header received
0925 07:21:04.492233 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.492234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.492246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.492300 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.492623 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.493145 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.493906 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.494113 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.494198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.495196 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.495198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.495199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.495207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.496663 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.497016 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.497018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.497018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.497078 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.497285 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.497286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.497286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.497396 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.497469 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.497639 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.497713 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:21:04.497833 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.491979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41236)
0925 07:21:04.492099 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.492109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.492170 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.492306 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.492306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.492488 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.492537 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.492539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.493241 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.493242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.493811 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.493813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.494200 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.494201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.494263 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.494751 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.494757 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.495059 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.496764 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.496765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.496766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.496944 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.497144 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.497145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.497146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.497204 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.497471 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.497472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.497519 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.497598 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.497707 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 07:21:04.498368 10411 tablet_copy_service.cc:140] P f596c959984e45428d3656d6bde12096: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 09572b9180d146049f26d59a7dd4ae2a ({username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:41236)
I20260925 07:21:04.498436 10411 tablet_copy_service.cc:161] P f596c959984e45428d3656d6bde12096: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 09572b9180d146049f26d59a7dd4ae2a at {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:41236: session id = 09572b9180d146049f26d59a7dd4ae2a-00000000000000000000000000000000
I20260925 07:21:04.498875 10411 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:21:04.499362 11042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 07:21:04.500427 11042 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: tablet copy: Starting download of 0 data blocks...
I20260925 07:21:04.500588 11042 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: tablet copy: Starting download of 1 WAL segments...
I20260925 07:21:04.501380 11042 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:21:04.502105 10411 tablet_copy_service.cc:342] P f596c959984e45428d3656d6bde12096: Request end of tablet copy session 09572b9180d146049f26d59a7dd4ae2a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:41236
I20260925 07:21:04.502164 10411 tablet_copy_service.cc:433] P f596c959984e45428d3656d6bde12096: ending tablet copy session 09572b9180d146049f26d59a7dd4ae2a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 09572b9180d146049f26d59a7dd4ae2a
I20260925 07:21:04.504364 11042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.505584 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.505823 11042 server_base.cc:1034] running on GCE node
W20260925 07:21:04.505821 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.512086 11159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.512322 11042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.512528 11042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.513656 11042 hybrid_clock.cc:648] HybridClock initialized: now 1790320864513655 us; error 38 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.515170 11042 init.cc:377] Logged in from keytab as oryx/127.4.47.60@KRBTEST.COM (short username oryx)
I20260925 07:21:04.515472 11042 webserver.cc:460] Webserver started at http://127.4.47.60:43091/ using document root <none> and password file <none>
I20260925 07:21:04.515594 11042 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.515630 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.516323 11042 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.516875 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.517026 11042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:04.517076 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "09572b9180d146049f26d59a7dd4ae2a"
format_stamp: "Formatted at 2026-09-25 07:21:04 on dist-test-slave-zw07"
server_key: "3c5c61ce3938b51e78b76fdc3d75170c"
server_key_iv: "b95d1273d7e4b76d0469c250cf45b652"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.517141 11042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.534670 11042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.536499 11042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.539682 11042 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.60:41083
I20260925 07:21:04.540102 11042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
I20260925 07:21:04.540366 11218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.60:41083 every 8 connection(s)
I20260925 07:21:04.540944 11219 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:21:04.541442 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Bootstrap starting.
I20260925 07:21:04.544094 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:21:04.544356 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Bootstrap complete.
I20260925 07:21:04.544567 11219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
I20260925 07:21:04.544661 11219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09572b9180d146049f26d59a7dd4ae2a, State: Initialized, Role: LEARNER
I20260925 07:21:04.544708 11219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } }
I20260925 07:21:04.544891 11219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:04.544884 11222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.545104 11222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: LEARNER
I20260925 07:21:04.546476 11233 catalog_manager.cc:1259] Loaded cluster ID: 5afab9184422469597fc0743cd344742
I20260925 07:21:04.546554 11233 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: loading cluster ID for follower catalog manager: success
I20260925 07:21:04.547809 11233 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: acquiring CA information for follower catalog manager: success
I20260925 07:21:04.548130 11233 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 07:21:04.592056 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:04.592371 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:04.592417 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:04.594291 11143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:04.594363 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:04.594410 11143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:04.596814 11143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:44947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=37205
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:04.597126 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:04.597421 11143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.603354 11238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.603397 11239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.603364 11241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.604038 11143 server_base.cc:1034] running on GCE node
I20260925 07:21:04.604267 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.604529 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.605700 11143 hybrid_clock.cc:648] HybridClock initialized: now 1790320864605688 us; error 34 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.608924 11143 init.cc:377] Logged in from keytab as oryx/127.4.47.1@KRBTEST.COM (short username oryx)
I20260925 07:21:04.609428 11143 webserver.cc:460] Webserver started at http://127.4.47.1:37205/ using document root <none> and password file <none>
I20260925 07:21:04.609684 11143 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.609773 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.611323 11143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:04.612008 11248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.612217 11143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:04.612326 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "7a2c51a42550414894f20a023560b8e2"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "e899b4a0d7823428e3c3d87964a4e9b0"
server_key_iv: "563161220cd18abbd46a8c1c7b67cb6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.612689 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.630751 11143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.633288 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:04.633477 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.633744 11143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:04.634079 11143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:04.634135 11143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.634177 11143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:04.634202 11143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.640383 11143 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:44947
I20260925 07:21:04.640434 11361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:44947 every 8 connection(s)
I20260925 07:21:04.640785 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:04.647946 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11143
I20260925 07:21:04.648101 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10587
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
I20260925 07:21:04.654075 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.641513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:47205 (local address 127.4.47.60:41083)
0925 07:21:04.641802 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.641804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.641992 (+ 188us) server_negotiation.cc:407] Connection header received
0925 07:21:04.642457 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.642458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.642472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.642521 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.643297 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.644232 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645257 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645638 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645707 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.649933 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.649935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.649936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.649944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.651852 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.652857 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.652859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.652859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.652928 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.653365 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.653366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.653366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.653497 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.653539 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.653882 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.653973 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":54,"threads_started":1}
I20260925 07:21:04.654074 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.642004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:34423 (local address 127.4.47.62:45079)
0925 07:21:04.642057 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.642060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.642069 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:04.642471 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.642473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.642486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.642521 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.643259 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.643774 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645350 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645637 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645707 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.649635 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.649637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.649637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.649646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.651902 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.652698 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.652700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.652700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.652770 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.653276 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.653277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.653277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.653433 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.653477 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.653883 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.653972 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:21:04.654505 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:34957
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:45079
--webserver_port=37269
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:21:04.654728 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.641829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:34423)
0925 07:21:04.642017 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.642026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.642380 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.642620 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.642621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.642922 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.643147 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.643154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.643897 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.643898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645213 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.645215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645709 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645866 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.646648 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.646655 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.649501 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.651980 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.651980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.651981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.652622 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.653079 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.653080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.653080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.653180 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.653527 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.653533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.653712 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.654243 (+ 531us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.654450 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":38,"thread_start_us":54,"threads_started":1}
I20260925 07:21:04.655220 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.641531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.4.47.1:47205)
0925 07:21:04.641927 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.641939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.642360 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.643007 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.643007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.643177 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.643235 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.643236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.644311 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.644312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645181 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.645183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645820 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645868 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.646584 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.646589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.649507 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.651955 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.651963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.651965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.652407 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.653024 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.653025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.653026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.653287 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.653580 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.653581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.653712 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.654974 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.655110 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260925 07:21:04.655238 11363 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:04.655457 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.641513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:56629 (local address 127.4.47.61:40589)
0925 07:21:04.641561 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.641563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.641831 (+ 268us) server_negotiation.cc:407] Connection header received
0925 07:21:04.642456 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.642457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.642469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.642525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.643237 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.644034 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645327 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645526 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645566 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.652006 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.652008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.652008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.652017 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.653773 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.654161 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.654162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.654162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.654216 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.654437 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.654438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.654438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.654598 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.654648 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.655216 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.655316 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:21:04.655550 11363 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.655767 11363 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:04.655978 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.641531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.4.47.1:56629)
0925 07:21:04.641698 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.641736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.642334 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.642601 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.642604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.642896 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.643146 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.643151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.644125 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.644125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645210 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.645213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.645700 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.645701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.645868 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.646563 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.646575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.651886 (+ 5311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.653863 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.653864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.653864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.654084 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.654306 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.654307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.654307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.654368 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.654696 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.654697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.654755 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.655535 (+ 780us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.655652 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"spinlock_wait_cycles":7936,"thread_start_us":96,"threads_started":1}
I20260925 07:21:04.656152 11362 heartbeater.cc:347] Connected to a master server at 127.4.47.60:41083
I20260925 07:21:04.656196 11362 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.656250 11362 heartbeater.cc:511] Master 127.4.47.60:41083 requested a full tablet report, sending...
I20260925 07:21:04.656348 11364 heartbeater.cc:347] Connected to a master server at 127.4.47.61:40589
I20260925 07:21:04.656399 11364 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.656464 11364 heartbeater.cc:511] Master 127.4.47.61:40589 requested a full tablet report, sending...
I20260925 07:21:04.656538 10391 ts_manager.cc:194] Re-registered known tserver with Master: 7a2c51a42550414894f20a023560b8e2 (127.4.47.1:44947)
I20260925 07:21:04.656634 11183 ts_manager.cc:194] Registered new tserver with Master: 7a2c51a42550414894f20a023560b8e2 (127.4.47.1:44947)
I20260925 07:21:04.656803 10988 ts_manager.cc:194] Registered new tserver with Master: 7a2c51a42550414894f20a023560b8e2 (127.4.47.1:44947)
I20260925 07:21:04.657022 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:34423
W20260925 07:21:04.742262 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:04.742503 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:04.742544 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:04.743767 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:04.743834 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:04.743880 11370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:04.745577 11370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:34957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=37269
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:04.745863 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:04.746119 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.751231 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.751231 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.751262 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.751483 11370 server_base.cc:1034] running on GCE node
I20260925 07:21:04.751677 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.751924 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.753084 11370 hybrid_clock.cc:648] HybridClock initialized: now 1790320864753071 us; error 24 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.755707 11370 init.cc:377] Logged in from keytab as oryx/127.4.47.2@KRBTEST.COM (short username oryx)
I20260925 07:21:04.756167 11370 webserver.cc:460] Webserver started at http://127.4.47.2:37269/ using document root <none> and password file <none>
I20260925 07:21:04.756349 11370 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.756409 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.757700 11370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:04.758252 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.758437 11370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.758526 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "35f972d202cc4affb758f28baf1b014d"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "a3a5469375162837243689243447a253"
server_key_iv: "12a45401e8b31371a1c6b026e2f0afbf"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.758849 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.777251 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.779261 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:04.779379 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.779600 11370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:04.779937 11370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:04.779971 11370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.779994 11370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:04.780043 11370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.785861 11370 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:34957
I20260925 07:21:04.785967 11498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:34957 every 8 connection(s)
I20260925 07:21:04.786294 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:04.790624 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11370
I20260925 07:21:04.790788 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10721
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
I20260925 07:21:04.797641 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:59703 (local address 127.4.47.62:45079)
0925 07:21:04.787232 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.787234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.787427 (+ 193us) server_negotiation.cc:407] Connection header received
0925 07:21:04.788551 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.788553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.788565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.788621 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.789665 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.790183 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791494 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.791817 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791892 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.794416 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.794417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.794417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.794426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.796134 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.796629 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.796630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.796630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.796690 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.796979 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.796980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.796980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.797111 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.797188 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.797465 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.797546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:21:04.798166 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:59703)
0925 07:21:04.787298 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.787308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.788435 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.788697 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.788701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.789177 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.789446 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.789451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.790296 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.790297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.791394 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.791397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.791951 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.792060 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.792748 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.792752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.794306 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.796226 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.796240 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.796243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.796548 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.796771 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.796772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.796773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.796890 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.797180 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.797186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.797332 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.797444 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.797604 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260925 07:21:04.798336 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:46745
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:45079
--webserver_port=35979
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:21:04.799115 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.4.47.2:50821)
0925 07:21:04.787549 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.787560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.788461 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.788700 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.788701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.789225 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.789449 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.789452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.790442 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.790443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.791424 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.791427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.791921 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.792062 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.792663 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.792673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.794692 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.797918 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.797919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.797919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.798147 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.798400 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.798401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.798401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.798473 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.798766 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.798767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.798807 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.798877 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.798965 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":49,"thread_start_us":59,"threads_started":1}
I20260925 07:21:04.799209 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:50821 (local address 127.4.47.60:41083)
0925 07:21:04.787553 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.787555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.787605 (+ 50us) server_negotiation.cc:407] Connection header received
0925 07:21:04.788563 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.788565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.788581 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.788612 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.789558 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.790332 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791529 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.791832 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.791894 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.794788 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.794790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.794790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.794798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.797841 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.798261 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.798262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.798262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.798320 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.798553 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.798554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.798555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.798676 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.798718 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.798933 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.799038 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
I20260925 07:21:04.800081 11499 heartbeater.cc:347] Connected to a master server at 127.4.47.60:41083
I20260925 07:21:04.800102 11500 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:04.800204 11499 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.800204 11500 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.800420 11500 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:04.800420 11499 heartbeater.cc:511] Master 127.4.47.60:41083 requested a full tablet report, sending...
I20260925 07:21:04.800925 10391 ts_manager.cc:194] Re-registered known tserver with Master: 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957)
I20260925 07:21:04.800936 11183 ts_manager.cc:194] Registered new tserver with Master: 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957)
I20260925 07:21:04.801501 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:59703
I20260925 07:21:04.803056 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:33521 (local address 127.4.47.61:40589)
0925 07:21:04.787108 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.787111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.787405 (+ 294us) server_negotiation.cc:407] Connection header received
0925 07:21:04.788607 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.788608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.788621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.788670 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.789558 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.790381 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.797344 (+ 6963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.797664 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.797739 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.800282 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.800284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.800284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.800292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.801938 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.802310 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.802311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.802311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.802373 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.802556 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.802557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.802557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.802699 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.802770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.802936 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.802997 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:21:04.803484 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.787079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.4.47.2:33521)
0925 07:21:04.787255 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.787297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.788461 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.788710 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.788711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.789188 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.789449 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.789452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.796130 (+ 6678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.796131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.797083 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.797086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.797834 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.797835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.797919 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.798401 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.798406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.800152 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.802008 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.802009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.802009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.802220 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.802437 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.802438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.802438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.802488 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.802780 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.802781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.802824 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.802881 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.802998 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":70,"thread_start_us":84,"threads_started":1}
I20260925 07:21:04.803916 11501 heartbeater.cc:347] Connected to a master server at 127.4.47.61:40589
I20260925 07:21:04.803979 11501 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.804054 11501 heartbeater.cc:511] Master 127.4.47.61:40589 requested a full tablet report, sending...
I20260925 07:21:04.804545 10988 ts_manager.cc:194] Registered new tserver with Master: 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957)
W20260925 07:21:04.884733 11506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:04.884949 11506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:04.884974 11506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:04.886127 11506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:04.886171 11506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:04.886199 11506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:04.887746 11506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:46745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=35979
--tserver_master_addrs=127.4.47.62:45079,127.4.47.61:40589,127.4.47.60:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:04.888015 11506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:04.888271 11506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:04.893415 11511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.893491 11512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:04.893415 11514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:04.893540 11506 server_base.cc:1034] running on GCE node
I20260925 07:21:04.893800 11506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:04.894065 11506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:04.895202 11506 hybrid_clock.cc:648] HybridClock initialized: now 1790320864895178 us; error 36 us; skew 500 ppm
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:04.897926 11506 init.cc:377] Logged in from keytab as oryx/127.4.47.3@KRBTEST.COM (short username oryx)
I20260925 07:21:04.898430 11506 webserver.cc:460] Webserver started at http://127.4.47.3:35979/ using document root <none> and password file <none>
I20260925 07:21:04.898674 11506 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:04.898732 11506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:04.900326 11506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:04.901031 11521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.901320 11506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:04.901408 11506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "e124957559204d0ea240a539dd2d1c4f"
format_stamp: "Formatted at 2026-09-25 07:21:03 on dist-test-slave-zw07"
server_key: "6ccf019378d5da4920c4f20f8fd59320"
server_key_iv: "e6344c72eb15aabd934fec4cd7626dd0"
server_key_version: "encryptionkey@0"
I20260925 07:21:04.901785 11506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:04.911020 11506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:04.913107 11506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:04.913254 11506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:04.913508 11506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:04.913816 11506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:04.913884 11506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.913925 11506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:04.913954 11506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:04.920022 11506 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:46745
I20260925 07:21:04.920071 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:46745 every 8 connection(s)
I20260925 07:21:04.920428 11506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:04.926402 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11506
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.61@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.60@KRBTEST.COM
Sep 25 07:21:04 dist-test-slave-zw07 krb5kdc[10346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320864, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:04.932670 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.4.47.3:58957)
0925 07:21:04.921370 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.921407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.922046 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.922317 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.922322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.922888 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.923115 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.923122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.923849 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.923850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.924821 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.924823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925523 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.925608 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.926059 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.926070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.929233 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.930999 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.931362 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.931572 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.931697 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.931945 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.931952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.932124 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.932226 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.932410 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":44,"thread_start_us":103,"threads_started":1}
I20260925 07:21:04.933030 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.4.47.3:60923)
0925 07:21:04.921494 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.921504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.922046 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.922603 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.922604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.922873 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.923116 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.923123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.924672 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.924673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.925347 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.925348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925758 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.925829 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.926180 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.926184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.928205 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.931190 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.931619 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.931879 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.931951 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.932309 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.932310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.932350 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.932420 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.932508 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260925 07:21:04.933048 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:58957 (local address 127.4.47.60:41083)
0925 07:21:04.921241 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.921244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.921541 (+ 297us) server_negotiation.cc:407] Connection header received
0925 07:21:04.922169 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.922171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.922184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.922224 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.923233 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.923724 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925215 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925444 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925508 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.929342 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.929343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.929344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.929352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.930909 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931448 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.931449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.931449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.931506 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931762 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.931763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.931763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.931895 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.931957 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.932844 (+ 887us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.932939 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:04.933398 11635 heartbeater.cc:347] Connected to a master server at 127.4.47.60:41083
I20260925 07:21:04.933509 11635 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.933593 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:60923 (local address 127.4.47.61:40589)
0925 07:21:04.921242 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.921244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.921564 (+ 320us) server_negotiation.cc:407] Connection header received
0925 07:21:04.922155 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.922157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.922169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.922224 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.923829 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.924574 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925433 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925679 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925759 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.928291 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.928293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.928294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.928302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.931097 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931720 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.931721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.931721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.931796 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.932018 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.932019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.932019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.932143 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.932204 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.933335 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.933418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:04.933715 11635 heartbeater.cc:511] Master 127.4.47.60:41083 requested a full tablet report, sending...
I20260925 07:21:04.933919 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:57877 (local address 127.4.47.62:45079)
0925 07:21:04.921721 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.921723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.921732 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:04.922170 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.922171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.922184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.922224 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.923203 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.923711 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.924909 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925114 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925165 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.929542 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.929543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.929543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.929550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.931190 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931760 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.931761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.931761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.931816 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.933363 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.933364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.933365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.933493 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.933537 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.933770 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.933849 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:21:04.934168 11183 ts_manager.cc:194] Registered new tserver with Master: e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:04.934259 11637 heartbeater.cc:347] Connected to a master server at 127.4.47.61:40589
I20260925 07:21:04.934311 11637 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.934381 11637 heartbeater.cc:511] Master 127.4.47.61:40589 requested a full tablet report, sending...
I20260925 07:21:04.934646 10988 ts_manager.cc:194] Registered new tserver with Master: e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:04.935285 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.921488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:57877)
0925 07:21:04.921649 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.921658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.922047 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.922321 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.922322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.922868 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:04.923115 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.923122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.923840 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.923841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.924811 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.924813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.925194 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.925195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.925329 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.927398 (+ 2069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:04.927403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:04.929443 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:04.931281 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.931656 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.931890 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:04.931890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:04.931891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:04.933272 (+ 1381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:04.933587 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:04.933588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:04.933628 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.934213 (+ 585us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.934890 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":47,"thread_start_us":60,"threads_started":1}
I20260925 07:21:04.936204 11636 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45079
I20260925 07:21:04.936259 11636 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:04.936321 11636 heartbeater.cc:511] Master 127.4.47.62:45079 requested a full tablet report, sending...
I20260925 07:21:04.936522 10391 ts_manager.cc:194] Re-registered known tserver with Master: e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:04.936985 10391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:57877
I20260925 07:21:04.937748 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.927760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.4.47.60:41083)
0925 07:21:04.928687 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.928689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.929273 (+ 584us) server_negotiation.cc:407] Connection header received
0925 07:21:04.929418 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.929418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.929430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.929470 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.930105 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.930969 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.932054 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.932334 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.932385 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.934540 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.934541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.934542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.934548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.936442 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.936862 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.936863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.936863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.936918 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.937135 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.937136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.937136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.937265 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.937341 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.937560 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.937641 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":60,"threads_started":1}
I20260925 07:21:04.941145 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.927648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.4.47.62:45079)
0925 07:21:04.928464 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.928466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.928775 (+ 309us) server_negotiation.cc:407] Connection header received
0925 07:21:04.928835 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.928836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.928853 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.928932 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.932046 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.932742 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.935144 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.935469 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.935545 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.937767 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.937768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.937769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.937776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.939930 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.940348 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.940349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.940349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.940414 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.940655 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.940655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.940656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.940763 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.940838 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.940977 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.941036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":81,"threads_started":1}
I20260925 07:21:04.942440 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.928452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.4.47.61:40589)
0925 07:21:04.928884 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.928886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.928894 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:04.935168 (+ 6274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.935169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.935181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.935902 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:04.935937 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.936646 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.937659 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.937970 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.938064 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.939512 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:04.939514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:04.939514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:04.939520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:04.941207 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.941620 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.941621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.941621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.941676 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:04.941851 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:04.941852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:04.941852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:04.941970 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:04.942076 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.942161 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.942235 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":58,"threads_started":1}
I20260925 07:21:04.943150 10391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:04.944928 10391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:04.952966 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.4.47.3:46745)
0925 07:21:04.949582 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.949587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.949598 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:21:04.949645 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.949647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.949664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.949753 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:04.949883 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.950706 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.951997 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.952568 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.952676 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.952765 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.952841 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.952877 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260925 07:21:04.952957 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:46745 (local address 127.0.0.1:34198)
0925 07:21:04.949438 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.949449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.949514 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.949754 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.949755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.949757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:04.949800 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.949802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.950822 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.950823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.951898 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.951900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.952678 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.952679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.952743 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.952748 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.952825 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.952870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260925 07:21:04.953959 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:34957 (local address 127.0.0.1:50752)
0925 07:21:04.949066 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.949080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.949121 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.949929 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.949930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.949932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:04.949958 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.949960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.951387 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.951388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.952164 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.952166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.953743 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.953744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.953805 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.953808 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.953839 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.953879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 07:21:04.953843 11569 tablet_service.cc:1467] Processing CreateTablet for tablet 85419e53164940958eb0e7385f3400ae (DEFAULT_TABLE table=test-table [id=4afb2d85d8094d33af5389171ee5ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:04.954272 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:44947 (local address 127.0.0.1:56656)
0925 07:21:04.949672 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.949682 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.949732 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.950018 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.950019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.950021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:04.950066 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.950067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.951379 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.951380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.952809 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.952810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.953993 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.953994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.954065 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.954069 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.954137 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.954178 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":761,"thread_start_us":84,"threads_started":1}
I20260925 07:21:04.954285 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.4.47.1:44947)
0925 07:21:04.949884 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.949887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.949897 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:04.949932 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.949934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.949948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.950021 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:04.950121 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.951299 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.952906 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.953925 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.953992 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.954072 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.954153 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.954200 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":82,"threads_started":1}
I20260925 07:21:04.954299 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85419e53164940958eb0e7385f3400ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:04.954443 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.949151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.4.47.2:34957)
0925 07:21:04.949632 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.949783 (+ 151us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.949793 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:04.949838 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.949840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.949856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.949930 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:04.950145 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.951284 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.952942 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.953635 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.954129 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.954202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.954224 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.954262 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":111,"threads_started":1}
I20260925 07:21:04.955053 11296 tablet_service.cc:1467] Processing CreateTablet for tablet 85419e53164940958eb0e7385f3400ae (DEFAULT_TABLE table=test-table [id=4afb2d85d8094d33af5389171ee5ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:04.955055 11433 tablet_service.cc:1467] Processing CreateTablet for tablet 85419e53164940958eb0e7385f3400ae (DEFAULT_TABLE table=test-table [id=4afb2d85d8094d33af5389171ee5ea5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:04.955416 11296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85419e53164940958eb0e7385f3400ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:04.955422 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85419e53164940958eb0e7385f3400ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:04.957863 11661 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Bootstrap starting.
I20260925 07:21:04.958580 11661 tablet_bootstrap.cc:654] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:04.958925 11661 log.cc:824] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:04.959151 11662 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Bootstrap starting.
I20260925 07:21:04.959450 11663 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Bootstrap starting.
I20260925 07:21:04.959805 11661 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: No bootstrap required, opened a new log
I20260925 07:21:04.959852 11662 tablet_bootstrap.cc:654] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:04.959896 11661 ts_tablet_manager.cc:1397] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:21:04.959913 11663 tablet_bootstrap.cc:654] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:04.960119 11663 log.cc:824] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:04.960217 11662 log.cc:824] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:04.960894 11663 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: No bootstrap required, opened a new log
I20260925 07:21:04.960916 11662 tablet_bootstrap.cc:492] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: No bootstrap required, opened a new log
I20260925 07:21:04.960960 11662 ts_tablet_manager.cc:1397] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:21:04.960959 11663 ts_tablet_manager.cc:1397] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:21:04.962915 11661 raft_consensus.cc:348] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.963179 11661 raft_consensus.cc:374] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:04.963027 11662 raft_consensus.cc:348] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.963223 11661 raft_consensus.cc:729] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e124957559204d0ea240a539dd2d1c4f, State: Initialized, Role: FOLLOWER
I20260925 07:21:04.963238 11662 raft_consensus.cc:374] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:04.963282 11662 raft_consensus.cc:729] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2c51a42550414894f20a023560b8e2, State: Initialized, Role: FOLLOWER
I20260925 07:21:04.963368 11661 consensus_queue.cc:260] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.963415 11662 consensus_queue.cc:260] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.963656 11662 ts_tablet_manager.cc:1428] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:21:04.963651 11661 ts_tablet_manager.cc:1428] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 07:21:04.963819 11636 heartbeater.cc:503] Master 127.4.47.62:45079 was elected leader, sending a full tablet report...
I20260925 07:21:04.964905 11363 heartbeater.cc:503] Master 127.4.47.62:45079 was elected leader, sending a full tablet report...
I20260925 07:21:04.965085 11663 raft_consensus.cc:348] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.965268 11663 raft_consensus.cc:374] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:04.965317 11663 raft_consensus.cc:729] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f972d202cc4affb758f28baf1b014d, State: Initialized, Role: FOLLOWER
I20260925 07:21:04.965440 11663 consensus_queue.cc:260] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:04.965718 11663 ts_tablet_manager.cc:1428] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 07:21:04.966010 11500 heartbeater.cc:503] Master 127.4.47.62:45079 was elected leader, sending a full tablet report...
I20260925 07:21:04.969717 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.966428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.0.0.1:46962)
0925 07:21:04.966757 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:04.966768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:04.966852 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:04.967057 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:04.967058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:04.967091 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:04.967138 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.967140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.967988 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.967989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.968973 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:04.968975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.969531 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.969532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:04.969578 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.969581 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:04.969631 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:21:04.969667 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260925 07:21:04.969710 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:04.966566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.4.47.60:41083)
0925 07:21:04.966634 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:04.966637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:04.966803 (+ 166us) server_negotiation.cc:407] Connection header received
0925 07:21:04.966931 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:04.966932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:04.966952 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:04.967012 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:04.967217 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.967910 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.969041 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:04.969454 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:04.969515 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:04.969562 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:04.969636 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 07:21:04.969662 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:21:04.969969 11193 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 07:21:04.970335 11222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f596c959984e45428d3656d6bde12096. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:04.970420 11222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: LEARNER
W20260925 07:21:05.037145 11502 tablet.cc:2369] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:21:05.141763 11365 tablet.cc:2369] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:21:05.171229 11638 tablet.cc:2369] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:21:05.178139 11668 raft_consensus.cc:482] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:05.178256 11668 raft_consensus.cc:504] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:05.178612 11668 leader_election.cc:288] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957), e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:05.181655 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.179012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:35913 (local address 127.4.47.3:46745)
0925 07:21:05.179051 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.179053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.179089 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:21:05.179118 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.179118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.179130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.179192 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:05.179456 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.179937 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.180895 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.181299 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.181389 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.181399 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.181548 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.181585 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:21:05.181663 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.178890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:51763 (local address 127.4.47.2:34957)
0925 07:21:05.179051 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.179054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.179092 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:21:05.179118 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.179118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.179130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.179192 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:05.179376 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.179896 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.180926 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.181305 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.181392 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.181402 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.181552 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.181593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 07:21:05.181676 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.178851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:46745 (local address 127.4.47.1:35913)
0925 07:21:05.178977 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:05.178986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:05.179031 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:05.179258 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:05.179259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:05.179261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:05.179333 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:05.179335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.180026 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.180027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:05.180796 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:05.180798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.181389 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.181390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:05.181464 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.181467 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:05.181531 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:21:05.181578 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260925 07:21:05.181679 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.178819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:34957 (local address 127.4.47.1:51763)
0925 07:21:05.178956 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:05.178969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:05.179031 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:05.179217 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:05.179218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:05.179220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:05.179294 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:05.179296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.179991 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.179991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:05.180809 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:05.180811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.181392 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.181393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:05.181475 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.181478 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:05.181547 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:21:05.181593 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":63,"thread_start_us":62,"threads_started":1}
I20260925 07:21:05.182091 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85419e53164940958eb0e7385f3400ae" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e124957559204d0ea240a539dd2d1c4f" is_pre_election: true
I20260925 07:21:05.182296 11589 raft_consensus.cc:2454] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 0.
I20260925 07:21:05.182384 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85419e53164940958eb0e7385f3400ae" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f972d202cc4affb758f28baf1b014d" is_pre_election: true
I20260925 07:21:05.182546 11453 raft_consensus.cc:2454] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 0.
I20260925 07:21:05.182559 11252 leader_election.cc:302] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2c51a42550414894f20a023560b8e2, e124957559204d0ea240a539dd2d1c4f; no voters:
I20260925 07:21:05.182670 11668 raft_consensus.cc:2781] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:05.182749 11668 raft_consensus.cc:482] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:05.182780 11668 raft_consensus.cc:3037] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:05.183506 11668 raft_consensus.cc:504] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:05.183652 11668 leader_election.cc:288] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 election: Requested vote from peers 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957), e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:05.183791 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85419e53164940958eb0e7385f3400ae" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f972d202cc4affb758f28baf1b014d"
I20260925 07:21:05.183786 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85419e53164940958eb0e7385f3400ae" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e124957559204d0ea240a539dd2d1c4f"
I20260925 07:21:05.183857 11453 raft_consensus.cc:3037] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:05.183858 11589 raft_consensus.cc:3037] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:05.184710 11453 raft_consensus.cc:2454] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 1.
I20260925 07:21:05.184710 11589 raft_consensus.cc:2454] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 1.
I20260925 07:21:05.184892 11250 leader_election.cc:302] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f972d202cc4affb758f28baf1b014d, 7a2c51a42550414894f20a023560b8e2; no voters:
I20260925 07:21:05.185005 11668 raft_consensus.cc:2781] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:05.185132 11668 raft_consensus.cc:686] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 7a2c51a42550414894f20a023560b8e2, State: Running, Role: LEADER
I20260925 07:21:05.185231 11668 consensus_queue.cc:237] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } }
I20260925 07:21:05.186229 10391 catalog_manager.cc:5495] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a2c51a42550414894f20a023560b8e2 (127.4.47.1). New cstate: current_term: 1 leader_uuid: "7a2c51a42550414894f20a023560b8e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } health_report { overall_health: UNKNOWN } } }
I20260925 07:21:05.186971 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: NON_VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: true } } }
I20260925 07:21:05.187104 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:05.188046 10950 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: attempting to promote NON_VOTER 09572b9180d146049f26d59a7dd4ae2a to VOTER
I20260925 07:21:05.188176 10950 consensus_queue.cc:237] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:05.188583 11193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 LEARNER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 07:21:05.188644 10998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Refusing update from remote peer f596c959984e45428d3656d6bde12096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 07:21:05.188864 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 07:21:05.188967 11136 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 07:21:05.189642 11136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.189783 11193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.189815 10998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190578 11222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190658 11222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:21:05.190624 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190709 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:05.190737 11680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190779 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190822 11680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:21:05.190840 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:05.190905 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f596c959984e45428d3656d6bde12096" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:05.190972 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:05.239566 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.237345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.4.47.1:44947)
0925 07:21:05.237388 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.237390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.237402 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:21:05.237451 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.237452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.237471 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.237535 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:05.237675 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.238146 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.238893 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.239094 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.239145 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.239289 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:05.239393 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:05.239452 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.239490 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.239507 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:21:05.241379 11589 raft_consensus.cc:1261] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Refusing update from remote peer 7a2c51a42550414894f20a023560b8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:05.241393 11453 raft_consensus.cc:1261] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Refusing update from remote peer 7a2c51a42550414894f20a023560b8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:05.241767 11668 consensus_queue.cc:1035] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:05.241916 11676 consensus_queue.cc:1035] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:05.244602 11683 mvcc.cc:205] Tried to move back new op lower bound from 7333154264025939968 to 7333154263799255040. Current Snapshot: MvccSnapshot[applied={T|T < 7333154264025939968}]
I20260925 07:21:05.246332 11685 mvcc.cc:205] Tried to move back new op lower bound from 7333154264025939968 to 7333154263799255040. Current Snapshot: MvccSnapshot[applied={T|T < 7333154264025939968 or (T in {7333154264025939968})}]
I20260925 07:21:05.248171 10391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:05.248268 10391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:05.249622 10391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f596c959984e45428d3656d6bde12096: Sending DeleteTablet for 3 replicas of tablet 85419e53164940958eb0e7385f3400ae
I20260925 07:21:05.249953 11569 tablet_service.cc:1514] Processing DeleteTablet for tablet 85419e53164940958eb0e7385f3400ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:05 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:34198
I20260925 07:21:05.249950 11296 tablet_service.cc:1514] Processing DeleteTablet for tablet 85419e53164940958eb0e7385f3400ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:05 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:56656
I20260925 07:21:05.250164 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet 85419e53164940958eb0e7385f3400ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:05 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:50752
I20260925 07:21:05.250393 11695 tablet_replica.cc:330] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: stopping tablet replica
I20260925 07:21:05.250532 11695 raft_consensus.cc:2229] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:05.250588 11694 tablet_replica.cc:330] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: stopping tablet replica
I20260925 07:21:05.250700 11694 raft_consensus.cc:2229] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:05.250731 11695 raft_consensus.cc:2258] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:05.250816 11694 raft_consensus.cc:2258] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:05.251389 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10359
I20260925 07:21:05.251879 11695 ts_tablet_manager.cc:1905] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:05.253686 11695 ts_tablet_manager.cc:1918] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:05.253837 11695 log.cc:1196] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/85419e53164940958eb0e7385f3400ae
I20260925 07:21:05.254190 11695 ts_tablet_manager.cc:1939] T 85419e53164940958eb0e7385f3400ae P 35f972d202cc4affb758f28baf1b014d: Deleting consensus metadata
I20260925 07:21:05.251319 11694 ts_tablet_manager.cc:1905] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:21:05.257090 11387 connection.cc:535] server connection from 127.0.0.1:50752 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:21:05.258695 11694 ts_tablet_manager.cc:1918] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:05.258775 11694 log.cc:1196] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/85419e53164940958eb0e7385f3400ae
I20260925 07:21:05.259114 11694 ts_tablet_manager.cc:1939] T 85419e53164940958eb0e7385f3400ae P e124957559204d0ea240a539dd2d1c4f: Deleting consensus metadata
W20260925 07:21:05.259543 11525 connection.cc:441] server connection from 127.0.0.1:34198 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34198 (request call id 1) could send its response
I20260925 07:21:05.259585 11693 tablet_replica.cc:330] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: stopping tablet replica
I20260925 07:21:05.259697 11693 raft_consensus.cc:2229] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:05.259865 11693 raft_consensus.cc:2258] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:05.260432 11693 ts_tablet_manager.cc:1905] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:05.262171 11693 ts_tablet_manager.cc:1918] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:05.262243 11693 log.cc:1196] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/85419e53164940958eb0e7385f3400ae
I20260925 07:21:05.262555 11693 ts_tablet_manager.cc:1939] T 85419e53164940958eb0e7385f3400ae P 7a2c51a42550414894f20a023560b8e2: Deleting consensus metadata
W20260925 07:21:05.262897 11249 connection.cc:441] server connection from 127.0.0.1:56656 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56656 (request call id 1) could send its response
I20260925 07:21:05.267596 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.259145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.4.47.60:41083)
0925 07:21:05.259189 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.259191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.259485 (+ 294us) server_negotiation.cc:407] Connection header received
0925 07:21:05.259746 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.259747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.259759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.259789 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:05.260619 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.261386 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.262457 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.262775 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.262824 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.264531 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:05.264532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:05.264532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:05.264539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:05.266203 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.266616 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.266618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.266618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.266678 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.266923 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.266924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.266925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.267059 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:05.267165 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.267395 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.267498 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:05.268375 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.258961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.4.47.61:40589)
0925 07:21:05.259007 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.259009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.259436 (+ 427us) server_negotiation.cc:407] Connection header received
0925 07:21:05.259547 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.259548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.259566 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.259598 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:05.260915 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.261695 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.262954 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.263296 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.263338 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.264907 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:05.264909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:05.264909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:05.264917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:05.266837 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.267288 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.267289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.267289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.267345 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.267554 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.267555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.267555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.267678 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:05.267746 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.268186 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.268270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:05.270332 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:05.250988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.4.47.60:41083)
0925 07:21:05.251027 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:05.251029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:05.263138 (+ 12109us) server_negotiation.cc:407] Connection header received
0925 07:21:05.263275 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:05.263276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:05.263287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:05.263315 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:05.264056 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.264766 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.265838 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:05.266152 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:05.266194 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:05.267749 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:05.267750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:05.267751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:05.267756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:05.269286 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.269579 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.269580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.269580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.269641 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:05.269834 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:05.269835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:05.269835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:05.269956 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:05.270047 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:05.270161 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 07:21:05.270242 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:21:06.246148 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.245869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:35463)
0925 07:21:06.246007 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.246087 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260925 07:21:06.246430 11636 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.247117 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.246813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:56073)
0925 07:21:06.246938 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.247027 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":46,"threads_started":1}
W20260925 07:21:06.247335 11363 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.247437 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.247136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:49571)
0925 07:21:06.247310 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.247375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
W20260925 07:21:06.247617 11500 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.749914 11707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f596c959984e45428d3656d6bde12096)
I20260925 07:21:06.750069 11707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:06.750346 11707 leader_election.cc:288] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60:41083)
I20260925 07:21:06.750373 11708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f596c959984e45428d3656d6bde12096)
I20260925 07:21:06.750514 11708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:06.750661 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.750458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41722)
0925 07:21:06.750556 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.750597 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":62,"threads_started":1}
W20260925 07:21:06.750799 10973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.751092 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.750836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41732)
0925 07:21:06.750975 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.751022 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260925 07:21:06.751204 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.751120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41736)
0925 07:21:06.751162 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.751183 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 07:21:06.751293 10973 leader_election.cc:334] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
W20260925 07:21:06.751363 11168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.751544 11708 leader_election.cc:288] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 1e009b31a67c41ad85a684300be2a398 (127.4.47.61:40589)
I20260925 07:21:06.751849 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.751734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41748)
0925 07:21:06.751782 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.751821 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:21:06.751922 11168 leader_election.cc:334] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.753916 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.750776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.4.47.60:41083)
0925 07:21:06.750865 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:06.750868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:06.751017 (+ 149us) server_negotiation.cc:407] Connection header received
0925 07:21:06.751156 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:06.751157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:06.751169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:06.751207 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:06.751471 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.752240 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753184 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.753561 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:06.753667 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:06.753816 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:21:06.753854 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":44,"threads_started":1}
I20260925 07:21:06.754065 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.751095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:40589 (local address 127.0.0.1:60356)
0925 07:21:06.751317 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.751328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:06.751435 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:06.751816 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:06.751816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:06.751819 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:06.751853 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:06.751855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.752529 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.752531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:06.753256 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:06.753258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.753815 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:06.753885 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:06.753889 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:06.753964 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:21:06.753999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":39,"threads_started":1}
I20260925 07:21:06.754065 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.751205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.4.47.61:40589)
0925 07:21:06.751630 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:06.751632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:06.751639 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:06.751659 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:06.751659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:06.751669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:06.751779 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:06.751952 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.752439 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753350 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.753720 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753813 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:06.753823 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:06.753972 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:21:06.754007 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":85,"threads_started":1}
I20260925 07:21:06.753939 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.750623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:41083 (local address 127.0.0.1:46990)
0925 07:21:06.750709 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.750964 (+ 255us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:06.751047 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:06.751284 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:06.751285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:06.751287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:06.751344 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:06.751346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.752327 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.752328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:06.753089 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:06.753091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:06.753660 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:06.753661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:06.753730 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:06.753734 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:06.753808 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:21:06.753854 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":38,"threads_started":1}
I20260925 07:21:06.754266 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09572b9180d146049f26d59a7dd4ae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e009b31a67c41ad85a684300be2a398" is_pre_election: true
I20260925 07:21:06.754377 10998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09572b9180d146049f26d59a7dd4ae2a in term 1.
I20260925 07:21:06.754413 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e009b31a67c41ad85a684300be2a398" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09572b9180d146049f26d59a7dd4ae2a" is_pre_election: true
I20260925 07:21:06.754503 11193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e009b31a67c41ad85a684300be2a398 in term 1.
I20260925 07:21:06.754559 11167 leader_election.cc:302] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09572b9180d146049f26d59a7dd4ae2a, 1e009b31a67c41ad85a684300be2a398; no voters: f596c959984e45428d3656d6bde12096
I20260925 07:21:06.754631 10973 leader_election.cc:302] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09572b9180d146049f26d59a7dd4ae2a, 1e009b31a67c41ad85a684300be2a398; no voters: f596c959984e45428d3656d6bde12096
I20260925 07:21:06.754680 11708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 07:21:06.754714 11708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader f596c959984e45428d3656d6bde12096)
I20260925 07:21:06.754719 11707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 07:21:06.754732 11708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:21:06.754750 11707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f596c959984e45428d3656d6bde12096)
I20260925 07:21:06.754776 11707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:21:06.755462 11708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:06.755486 11707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:06.755584 11708 leader_election.cc:288] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 election: Requested vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 1e009b31a67c41ad85a684300be2a398 (127.4.47.61:40589)
I20260925 07:21:06.755739 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09572b9180d146049f26d59a7dd4ae2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e009b31a67c41ad85a684300be2a398"
I20260925 07:21:06.755789 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.755663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41770)
0925 07:21:06.755718 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.755754 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:21:06.755851 10998 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 09572b9180d146049f26d59a7dd4ae2a in current term 2: Already voted for candidate 1e009b31a67c41ad85a684300be2a398 in this term.
I20260925 07:21:06.755897 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.755663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41762)
0925 07:21:06.755848 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.755875 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 07:21:06.756001 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e009b31a67c41ad85a684300be2a398" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09572b9180d146049f26d59a7dd4ae2a"
I20260925 07:21:06.756095 11193 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e009b31a67c41ad85a684300be2a398 in current term 2: Already voted for candidate 09572b9180d146049f26d59a7dd4ae2a in this term.
I20260925 07:21:06.756209 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.756089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41776)
0925 07:21:06.756140 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.756167 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:21:06.756222 11707 leader_election.cc:288] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 election: Requested vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 09572b9180d146049f26d59a7dd4ae2a (127.4.47.60:41083)
W20260925 07:21:06.756280 10973 leader_election.cc:334] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.756311 10973 leader_election.cc:302] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e009b31a67c41ad85a684300be2a398; no voters: 09572b9180d146049f26d59a7dd4ae2a, f596c959984e45428d3656d6bde12096
I20260925 07:21:06.756358 11707 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 07:21:06.756610 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:06.756522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:41788)
0925 07:21:06.756569 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:06.756592 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:21:06.756687 11168 leader_election.cc:334] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:06.756739 11168 leader_election.cc:302] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09572b9180d146049f26d59a7dd4ae2a; no voters: 1e009b31a67c41ad85a684300be2a398, f596c959984e45428d3656d6bde12096
I20260925 07:21:06.756819 11708 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 07:21:07.247089 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:07.246754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:56345)
0925 07:21:07.246959 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:07.247002 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260925 07:21:07.247910 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:07.247634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:50553)
0925 07:21:07.247801 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:07.247841 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260925 07:21:07.248140 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:07.247917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:39917)
0925 07:21:07.248065 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:07.248097 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260925 07:21:08.247833 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.247505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:36275)
0925 07:21:08.247710 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.247756 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
I20260925 07:21:08.248582 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.248349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:46533)
0925 07:21:08.248481 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.248519 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
I20260925 07:21:08.248673 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.248507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:44251)
0925 07:21:08.248596 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.248629 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":49,"threads_started":1}
I20260925 07:21:08.266276 11723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:08.266458 11723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:08.266677 11723 leader_election.cc:288] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 1e009b31a67c41ad85a684300be2a398 (127.4.47.61:40589)
I20260925 07:21:08.267009 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.266817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42018)
0925 07:21:08.266914 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.266958 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":57,"threads_started":1}
I20260925 07:21:08.266968 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09572b9180d146049f26d59a7dd4ae2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e009b31a67c41ad85a684300be2a398" is_pre_election: true
I20260925 07:21:08.267083 10998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09572b9180d146049f26d59a7dd4ae2a in term 2.
I20260925 07:21:08.267217 11167 leader_election.cc:302] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09572b9180d146049f26d59a7dd4ae2a, 1e009b31a67c41ad85a684300be2a398; no voters:
I20260925 07:21:08.267340 11723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 07:21:08.267390 11723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:08.267411 11723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:21:08.267452 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.267327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42026)
0925 07:21:08.267366 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.267399 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 07:21:08.267546 11168 leader_election.cc:334] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:08.268172 11723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:08.268311 11723 leader_election.cc:288] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 election: Requested vote from peers f596c959984e45428d3656d6bde12096 (127.4.47.62:45079), 1e009b31a67c41ad85a684300be2a398 (127.4.47.61:40589)
I20260925 07:21:08.268481 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.268386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42042)
0925 07:21:08.268425 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.268456 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 07:21:08.268469 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09572b9180d146049f26d59a7dd4ae2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e009b31a67c41ad85a684300be2a398"
I20260925 07:21:08.268540 10998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:21:08.268822 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.268738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42046)
0925 07:21:08.268785 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.268807 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:21:08.268919 11168 leader_election.cc:334] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
I20260925 07:21:08.269313 10998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09572b9180d146049f26d59a7dd4ae2a in term 3.
I20260925 07:21:08.269491 11167 leader_election.cc:302] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09572b9180d146049f26d59a7dd4ae2a, 1e009b31a67c41ad85a684300be2a398; no voters: f596c959984e45428d3656d6bde12096
I20260925 07:21:08.269627 11723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 07:21:08.269769 11723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [term 3 LEADER]: Becoming Leader. State: Replica: 09572b9180d146049f26d59a7dd4ae2a, State: Running, Role: LEADER
I20260925 07:21:08.269852 11723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } }
I20260925 07:21:08.270262 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09572b9180d146049f26d59a7dd4ae2a. Latest consensus state: current_term: 3 leader_uuid: "09572b9180d146049f26d59a7dd4ae2a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:08.270358 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:08.270521 11728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:08.270977 11728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:08.271034 11728 catalog_manager.cc:1259] Loaded cluster ID: 5afab9184422469597fc0743cd344742
I20260925 07:21:08.271051 11728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:08.271288 11728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:08.271386 11728 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Loaded TSK: 0
I20260925 07:21:08.271533 11728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:21:08.274103 11182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:08.275705 11182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:08.276638 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.276549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42068)
0925 07:21:08.276586 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.276613 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 07:21:08.276666 10998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [term 3 FOLLOWER]: Refusing update from remote peer 09572b9180d146049f26d59a7dd4ae2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 07:21:08.276906 11726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 07:21:08.276971 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.276860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.0.0.1:42072)
0925 07:21:08.276920 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.276947 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 07:21:08.277160 11168 consensus_peers.cc:487] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a -> Peer f596c959984e45428d3656d6bde12096 (127.4.47.62:45079): Couldn't send request to peer f596c959984e45428d3656d6bde12096. Status: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 07:21:08.277719 11730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09572b9180d146049f26d59a7dd4ae2a. Latest consensus state: current_term: 3 leader_uuid: "09572b9180d146049f26d59a7dd4ae2a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:08.277822 11730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:21:08.278283 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "09572b9180d146049f26d59a7dd4ae2a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:08.278363 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:08.278554 11730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "09572b9180d146049f26d59a7dd4ae2a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f596c959984e45428d3656d6bde12096" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45079 } } peers { permanent_uuid: "1e009b31a67c41ad85a684300be2a398" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 40589 } attrs { promote: false } } peers { permanent_uuid: "09572b9180d146049f26d59a7dd4ae2a" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 41083 } attrs { promote: false } } }
I20260925 07:21:08.278637 11730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e009b31a67c41ad85a684300be2a398 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:21:08.283109 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:44947 (local address 127.0.0.1:56666)
0925 07:21:08.280270 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.280278 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:08.280317 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:08.280665 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:08.280666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:08.280668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:08.280711 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.280713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.281345 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.281346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.282141 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.282142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.282860 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.282861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.282925 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.282928 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:08.282983 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:21:08.283024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260925 07:21:08.283111 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.4.47.1:44947)
0925 07:21:08.280541 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:08.280543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:08.280552 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:08.280578 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:08.280578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:08.280590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:08.280664 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:08.280767 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.281255 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.282234 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.282780 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.282862 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.282873 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:08.282990 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 07:21:08.283022 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":56,"threads_started":1}
I20260925 07:21:08.283478 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.4.47.2:34957)
0925 07:21:08.280176 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:08.280179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:08.280231 (+ 52us) server_negotiation.cc:407] Connection header received
0925 07:21:08.280299 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:08.280300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:08.280310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:08.280380 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:08.280574 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.281155 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.282534 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.283124 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.283170 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.283180 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:08.283360 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 07:21:08.283396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
I20260925 07:21:08.283476 11296 tablet_service.cc:1467] Processing CreateTablet for tablet 089e2149254a42d5b969aa3eb1c023ca (DEFAULT_TABLE table=test-table [id=86ecbd0333454072ab4390bff44acf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:08.283610 11296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e2149254a42d5b969aa3eb1c023ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:08.283701 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:34957 (local address 127.0.0.1:50760)
0925 07:21:08.280177 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.280187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:08.280247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:08.280382 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:08.280383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:08.280384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:08.280441 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.280443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.281256 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.281258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.282448 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.282449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.283233 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.283234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.283290 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.283293 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:08.283593 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 07:21:08.283639 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 07:21:08.284754 11741 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: Bootstrap starting.
I20260925 07:21:08.284834 11433 tablet_service.cc:1467] Processing CreateTablet for tablet 089e2149254a42d5b969aa3eb1c023ca (DEFAULT_TABLE table=test-table [id=86ecbd0333454072ab4390bff44acf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:08.284956 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e2149254a42d5b969aa3eb1c023ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:08.285141 11741 tablet_bootstrap.cc:654] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:08.285693 11741 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: No bootstrap required, opened a new log
I20260925 07:21:08.285738 11741 ts_tablet_manager.cc:1397] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:08.285852 11741 raft_consensus.cc:348] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.285912 11741 raft_consensus.cc:374] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:08.285931 11741 raft_consensus.cc:729] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2c51a42550414894f20a023560b8e2, State: Initialized, Role: FOLLOWER
I20260925 07:21:08.285979 11741 consensus_queue.cc:260] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.286134 11741 ts_tablet_manager.cc:1428] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:08.286433 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.286312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:38085)
0925 07:21:08.286358 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.286406 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:21:08.286525 11744 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Bootstrap starting.
I20260925 07:21:08.286899 11744 tablet_bootstrap.cc:654] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:08.287214 11362 heartbeater.cc:503] Master 127.4.47.60:41083 was elected leader, sending a full tablet report...
I20260925 07:21:08.287518 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.287051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:37869)
0925 07:21:08.287461 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.287490 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
I20260925 07:21:08.287542 11744 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: No bootstrap required, opened a new log
I20260925 07:21:08.287591 11744 ts_tablet_manager.cc:1397] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:08.287720 11744 raft_consensus.cc:348] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.287791 11744 raft_consensus.cc:374] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:08.287817 11744 raft_consensus.cc:729] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f972d202cc4affb758f28baf1b014d, State: Initialized, Role: FOLLOWER
I20260925 07:21:08.287835 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:46745 (local address 127.0.0.1:34206)
0925 07:21:08.280228 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.280239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:08.280294 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:08.281439 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:08.281439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:08.281441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:08.281513 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.281514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.282402 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.282403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.286717 (+ 4314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:08.286720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.287601 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.287603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:08.287668 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.287673 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:21:08.287705 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 07:21:08.287747 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260925 07:21:08.287861 11744 consensus_queue.cc:260] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.288053 11744 ts_tablet_manager.cc:1428] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:08.288268 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.280555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.4.47.3:46745)
0925 07:21:08.281312 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:08.281314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:08.281321 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:08.281344 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:08.281345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:08.281355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:08.281425 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:08.281589 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.282309 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.286846 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.287498 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.288106 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.288122 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:08.288143 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 07:21:08.288180 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":58,"threads_started":1}
I20260925 07:21:08.288321 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.288226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:42849)
0925 07:21:08.288260 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.288292 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 07:21:08.288496 11499 heartbeater.cc:503] Master 127.4.47.60:41083 was elected leader, sending a full tablet report...
I20260925 07:21:08.288612 11569 tablet_service.cc:1467] Processing CreateTablet for tablet 089e2149254a42d5b969aa3eb1c023ca (DEFAULT_TABLE table=test-table [id=86ecbd0333454072ab4390bff44acf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:08.288760 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e2149254a42d5b969aa3eb1c023ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:08.289757 11747 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Bootstrap starting.
I20260925 07:21:08.289947 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.289823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:47869)
0925 07:21:08.289889 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.289924 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 07:21:08.290087 11747 tablet_bootstrap.cc:654] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:08.290571 11747 tablet_bootstrap.cc:492] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: No bootstrap required, opened a new log
I20260925 07:21:08.290609 11747 ts_tablet_manager.cc:1397] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:08.290717 11747 raft_consensus.cc:348] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.290766 11747 raft_consensus.cc:374] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:08.290779 11747 raft_consensus.cc:729] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e124957559204d0ea240a539dd2d1c4f, State: Initialized, Role: FOLLOWER
I20260925 07:21:08.290835 11747 consensus_queue.cc:260] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.291049 11747 ts_tablet_manager.cc:1428] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:08.291316 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.291220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:60023)
0925 07:21:08.291261 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.291286 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 07:21:08.291495 11635 heartbeater.cc:503] Master 127.4.47.60:41083 was elected leader, sending a full tablet report...
I20260925 07:21:08.291778 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.291628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:51717)
0925 07:21:08.291724 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.291754 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 07:21:08.451798 11743 raft_consensus.cc:482] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:08.451930 11743 raft_consensus.cc:504] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.452155 11743 leader_election.cc:288] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957), e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:08.452484 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e2149254a42d5b969aa3eb1c023ca" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e124957559204d0ea240a539dd2d1c4f" is_pre_election: true
I20260925 07:21:08.452484 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e2149254a42d5b969aa3eb1c023ca" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f972d202cc4affb758f28baf1b014d" is_pre_election: true
I20260925 07:21:08.452590 11589 raft_consensus.cc:2454] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 0.
I20260925 07:21:08.452590 11453 raft_consensus.cc:2454] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 0.
I20260925 07:21:08.452788 11250 leader_election.cc:302] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f972d202cc4affb758f28baf1b014d, 7a2c51a42550414894f20a023560b8e2; no voters:
I20260925 07:21:08.452925 11743 raft_consensus.cc:2781] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:08.453014 11743 raft_consensus.cc:482] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:08.453043 11743 raft_consensus.cc:3037] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:08.453776 11743 raft_consensus.cc:504] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.453933 11743 leader_election.cc:288] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 election: Requested vote from peers 35f972d202cc4affb758f28baf1b014d (127.4.47.2:34957), e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745)
I20260925 07:21:08.454102 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e2149254a42d5b969aa3eb1c023ca" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e124957559204d0ea240a539dd2d1c4f"
I20260925 07:21:08.454121 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e2149254a42d5b969aa3eb1c023ca" candidate_uuid: "7a2c51a42550414894f20a023560b8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f972d202cc4affb758f28baf1b014d"
I20260925 07:21:08.454180 11453 raft_consensus.cc:3037] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:08.454200 11589 raft_consensus.cc:3037] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:08.454779 11589 raft_consensus.cc:2454] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 1.
I20260925 07:21:08.454938 11453 raft_consensus.cc:2454] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2c51a42550414894f20a023560b8e2 in term 1.
I20260925 07:21:08.454975 11252 leader_election.cc:302] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2c51a42550414894f20a023560b8e2, e124957559204d0ea240a539dd2d1c4f; no voters:
I20260925 07:21:08.455106 11743 raft_consensus.cc:2781] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:08.455256 11743 raft_consensus.cc:686] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 7a2c51a42550414894f20a023560b8e2, State: Running, Role: LEADER
I20260925 07:21:08.455329 11743 consensus_queue.cc:237] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } }
I20260925 07:21:08.455794 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.455679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:41073)
0925 07:21:08.455724 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.455757 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:21:08.456143 11182 catalog_manager.cc:5495] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a2c51a42550414894f20a023560b8e2 (127.4.47.1). New cstate: current_term: 1 leader_uuid: "7a2c51a42550414894f20a023560b8e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a2c51a42550414894f20a023560b8e2" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 44947 } health_report { overall_health: HEALTHY } } }
I20260925 07:21:08.465425 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.463153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.4.47.1:44947)
0925 07:21:08.463212 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:08.463214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:08.463222 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:08.463268 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:08.463269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:08.463282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:08.463353 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:08.463492 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.463974 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.464730 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:08.464940 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:08.465018 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:08.465164 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:08.465202 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:08.465264 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:08.465321 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 07:21:08.465342 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:08.466338 11453 raft_consensus.cc:1261] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Refusing update from remote peer 7a2c51a42550414894f20a023560b8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:08.466447 11589 raft_consensus.cc:1261] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Refusing update from remote peer 7a2c51a42550414894f20a023560b8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:08.466576 11743 consensus_queue.cc:1035] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f972d202cc4affb758f28baf1b014d" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:08.466678 11750 consensus_queue.cc:1035] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e124957559204d0ea240a539dd2d1c4f" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:08.467456 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.467151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:58285)
0925 07:21:08.467382 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.467420 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 07:21:08.467489 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.467351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:37789)
0925 07:21:08.467424 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.467457 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 07:21:08.468664 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.468521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.3:54227)
0925 07:21:08.468609 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.468636 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 07:21:08.468678 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.468502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:42001)
0925 07:21:08.468619 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.468649 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 07:21:08.469838 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.469751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.1:44797)
0925 07:21:08.469785 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.469808 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 07:21:08.470007 11183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:08.470116 11183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:08.470734 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.470560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45079 (local address 127.4.47.2:54823)
0925 07:21:08.470673 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.470700 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 07:21:08.471690 11183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09572b9180d146049f26d59a7dd4ae2a: Sending DeleteTablet for 3 replicas of tablet 089e2149254a42d5b969aa3eb1c023ca
I20260925 07:21:08.471943 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e2149254a42d5b969aa3eb1c023ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:08 UTC) from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:50760
I20260925 07:21:08.471980 11432 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e2149254a42d5b969aa3eb1c023ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:08 UTC) from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:50760
I20260925 07:21:08.472025 11569 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e2149254a42d5b969aa3eb1c023ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:08 UTC) from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:34206
I20260925 07:21:08.472128 11296 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e2149254a42d5b969aa3eb1c023ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:08 UTC) from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:56666
I20260925 07:21:08.472146 11295 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e2149254a42d5b969aa3eb1c023ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:08 UTC) from {username='oryx', principal='oryx/127.4.47.60@KRBTEST.COM'} at 127.0.0.1:56666
I20260925 07:21:08.472200 11763 tablet_replica.cc:330] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: stopping tablet replica
I20260925 07:21:08.472244 11763 raft_consensus.cc:2229] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:08.472281 11763 raft_consensus.cc:2258] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:08.472288 11764 tablet_replica.cc:330] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2: stopping tablet replica
I20260925 07:21:08.472335 11764 raft_consensus.cc:2229] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:08.472396 11764 raft_consensus.cc:2258] T 089e2149254a42d5b969aa3eb1c023ca P 7a2c51a42550414894f20a023560b8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:08.472440 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11143
I20260925 07:21:08.473045 11763 ts_tablet_manager.cc:1905] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:08.474926 11763 ts_tablet_manager.cc:1918] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:08.474985 11763 log.cc:1196] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/089e2149254a42d5b969aa3eb1c023ca
I20260925 07:21:08.475242 11763 ts_tablet_manager.cc:1939] T 089e2149254a42d5b969aa3eb1c023ca P e124957559204d0ea240a539dd2d1c4f: Deleting consensus metadata
I20260925 07:21:08.475762 11170 catalog_manager.cc:4841] TS e124957559204d0ea240a539dd2d1c4f (127.4.47.3:46745): tablet 089e2149254a42d5b969aa3eb1c023ca (table test-table [id=86ecbd0333454072ab4390bff44acf0e]) successfully deleted
I20260925 07:21:08.478611 11762 tablet_replica.cc:330] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: stopping tablet replica
I20260925 07:21:08.478693 11762 raft_consensus.cc:2229] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:08.478749 11762 raft_consensus.cc:2258] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:08.479017 11762 ts_tablet_manager.cc:1905] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:08.480518 11762 ts_tablet_manager.cc:1918] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:08.480581 11762 log.cc:1196] T 089e2149254a42d5b969aa3eb1c023ca P 35f972d202cc4affb758f28baf1b014d: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/089e2149254a42d5b969aa3eb1c023ca
I20260925 07:21:08.480618 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.480490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:44947 (local address 127.0.0.1:56678)
0925 07:21:08.480550 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.480582 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 07:21:08.480762 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11370
W20260925 07:21:08.480773 11168 catalog_manager.cc:4584] TS 7a2c51a42550414894f20a023560b8e2 (127.4.47.1:44947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 089e2149254a42d5b969aa3eb1c023ca: Network error: Client connection negotiation failed: client connection to 127.4.47.1:44947: connect: Connection refused (error 111)
I20260925 07:21:08.485878 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11506
I20260925 07:21:08.485908 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:08.485756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:34957 (local address 127.0.0.1:50770)
0925 07:21:08.485832 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:08.485866 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:34957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 07:21:08.490324 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 10855
I20260925 07:21:08.495234 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11042
2026-09-25T07:21:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6353 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:08 dist-test-slave-zw07 krb5kdc[11767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:08 dist-test-slave-zw07 krb5kdc[11767](info): set up 2 sockets
Sep 25 07:21:08 dist-test-slave-zw07 krb5kdc[11767](info): commencing operation
krb5kdc: starting...
W20260925 07:21:09.535694 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:09 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab.
Entry for principal oryx/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab.
I20260925 07:21:09.654175 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:44775
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab
--principal=oryx/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:09.736403 11780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:09.736649 11780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:09.736672 11780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:09.737898 11780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:09.737946 11780 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:21:09.737959 11780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:09.737972 11780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:09.738034 11780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:09.738051 11780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:09.739636 11780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:44775
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.62.keytab
--principal=oryx/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:09.739912 11780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:09.740161 11780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:09.745224 11786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:09.745241 11788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:09.745366 11785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:09.745536 11780 server_base.cc:1034] running on GCE node
I20260925 07:21:09.745761 11780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:09.746027 11780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:09.747133 11780 hybrid_clock.cc:648] HybridClock initialized: now 1790320869747120 us; error 28 us; skew 500 ppm
Sep 25 07:21:09 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:09.756376 11780 init.cc:377] Logged in from keytab as oryx/127.4.47.62@KRBTEST.COM (short username oryx)
I20260925 07:21:09.756903 11780 webserver.cc:460] Webserver started at http://127.4.47.62:45413/ using document root <none> and password file <none>
I20260925 07:21:09.757154 11780 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:09.757225 11780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:09.757421 11780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:09.758566 11780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "03078dcf10f04af4b2e4ad48baf084e7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "ee56e181e872fbf2f5e2b319d9c7bcec"
server_key_iv: "bc9d5f9b78e1f1a8fa659396dac1e783"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.758940 11780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "03078dcf10f04af4b2e4ad48baf084e7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "ee56e181e872fbf2f5e2b319d9c7bcec"
server_key_iv: "bc9d5f9b78e1f1a8fa659396dac1e783"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.760375 11780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:09.761199 11795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:09.761426 11780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:21:09.761543 11780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "03078dcf10f04af4b2e4ad48baf084e7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "ee56e181e872fbf2f5e2b319d9c7bcec"
server_key_iv: "bc9d5f9b78e1f1a8fa659396dac1e783"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.761621 11780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:09.795042 11780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:09.797297 11780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:09.797489 11780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:09.802531 11780 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:44775
I20260925 07:21:09.802578 11857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:44775 every 8 connection(s)
I20260925 07:21:09.802942 11780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:09.803622 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:09.805984 11858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Bootstrap starting.
I20260925 07:21:09.806587 11858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:09.806885 11858 log.cc:824] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:09.807603 11858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: No bootstrap required, opened a new log
I20260925 07:21:09.809296 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11780
I20260925 07:21:09.809373 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:09.809566 4284 external_mini_cluster.cc:1428] Setting key c47ccbabc258d1d8dfc89933f3ed96c6
I20260925 07:21:09.810051 11858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } }
I20260925 07:21:09.810258 11858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:09.810350 11858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03078dcf10f04af4b2e4ad48baf084e7, State: Initialized, Role: FOLLOWER
I20260925 07:21:09.810509 11858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } }
I20260925 07:21:09.810613 11858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:09.810658 11858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:09.810705 11858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:09.811316 11858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } }
I20260925 07:21:09.811432 11858 leader_election.cc:302] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03078dcf10f04af4b2e4ad48baf084e7; no voters:
I20260925 07:21:09.811627 11858 leader_election.cc:288] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:09.811692 11863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:09.811865 11858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:09.811887 11863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [term 1 LEADER]: Becoming Leader. State: Replica: 03078dcf10f04af4b2e4ad48baf084e7, State: Running, Role: LEADER
I20260925 07:21:09.812134 11863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } }
I20260925 07:21:09.812539 11864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03078dcf10f04af4b2e4ad48baf084e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } } }
I20260925 07:21:09.812619 11864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:09.812832 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03078dcf10f04af4b2e4ad48baf084e7. Latest consensus state: current_term: 1 leader_uuid: "03078dcf10f04af4b2e4ad48baf084e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03078dcf10f04af4b2e4ad48baf084e7" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 44775 } } }
I20260925 07:21:09.812963 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:09.813201 11872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:09.813773 11872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:09.815549 11872 catalog_manager.cc:1347] Generated new cluster ID: d78f09182d824c2ebbebdf14a7e2caa4
I20260925 07:21:09.815601 11872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:21:09 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:09.821280 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:09.810159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.4.47.62:44775)
0925 07:21:09.810370 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:09.810380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:09.810430 (+ 50us) server_negotiation.cc:407] Connection header received
0925 07:21:09.810589 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:09.810594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:09.810662 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:09.810803 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:09.811498 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.813004 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.814056 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.814656 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.814768 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:09.817637 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:09.817641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:09.817642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:09.817653 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:09.819556 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:09.819961 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:09.819964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:09.819965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:09.820083 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:09.820325 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:09.820326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:09.820326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:09.820625 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:09.820784 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:09.820912 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 07:21:09.821049 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260925 07:21:09.822154 11872 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:09.822634 11872 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:09.824453 11872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Generated new TSK 0
I20260925 07:21:09.824628 11872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
I20260925 07:21:09.853870 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:44775
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:09.959630 11884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:09.959892 11884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:09.959939 11884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:09.961370 11884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:09.961457 11884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:09.961494 11884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:21:09.961540 11884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:09.963284 11884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:09.963599 11884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:09.963876 11884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:09.970814 11892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:09.970877 11890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:09.970935 11889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:09.971109 11884 server_base.cc:1034] running on GCE node
I20260925 07:21:09.971307 11884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:09.971565 11884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:09.972710 11884 hybrid_clock.cc:648] HybridClock initialized: now 1790320869972697 us; error 28 us; skew 500 ppm
Sep 25 07:21:09 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:09.975208 11884 init.cc:377] Logged in from keytab as oryx/127.4.47.1@KRBTEST.COM (short username oryx)
I20260925 07:21:09.975677 11884 webserver.cc:460] Webserver started at http://127.4.47.1:37255/ using document root <none> and password file <none>
I20260925 07:21:09.975857 11884 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:09.975898 11884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:09.975991 11884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:09.977016 11884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "493c2941d8d01eaec2b4ce0f8297c16f"
server_key_iv: "162b31e77d9d0a4c2b905f99fc4dd3f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.977314 11884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "493c2941d8d01eaec2b4ce0f8297c16f"
server_key_iv: "162b31e77d9d0a4c2b905f99fc4dd3f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.978601 11884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:09.979303 11899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:09.979480 11884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:09.979571 11884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
format_stamp: "Formatted at 2026-09-25 07:21:09 on dist-test-slave-zw07"
server_key: "493c2941d8d01eaec2b4ce0f8297c16f"
server_key_iv: "162b31e77d9d0a4c2b905f99fc4dd3f8"
server_key_version: "encryptionkey@0"
I20260925 07:21:09.979635 11884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:09.988687 11884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:09.990985 11884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:09.991184 11884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:09.991705 11884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:09.991742 11884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:09.991770 11884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:09.991820 11884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:09.998580 11884 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:46583
I20260925 07:21:09.998689 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:46583 every 8 connection(s)
I20260925 07:21:09.999109 11884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:21:09 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.002964 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:09.992328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:36513 (local address 127.4.47.62:44775)
0925 07:21:09.992398 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:09.992401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:09.992633 (+ 232us) server_negotiation.cc:407] Connection header received
0925 07:21:09.993389 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:09.993390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:09.993409 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:09.993485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:09.994547 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.995454 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.997118 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.997495 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.997580 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.000435 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.000436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.000436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.000443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.001656 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.002069 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.002070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.002070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.002126 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.002374 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.002375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.002375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.002498 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.002551 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.002803 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.002897 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:10.003203 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:09.992281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.1:36513)
0925 07:21:09.992487 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:09.992525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:09.993251 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:09.993586 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:09.993590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:09.993994 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:09.994423 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:09.994431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.995571 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.995572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:09.997011 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:09.997014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:09.997600 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:09.997601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:09.997792 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:09.998459 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:09.998480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.000335 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.001734 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.001741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.001743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.001989 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.002192 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.002193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.002193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.002301 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.002561 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.002566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.002696 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.002790 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.002933 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260925 07:21:10.003767 12015 heartbeater.cc:347] Connected to a master server at 127.4.47.62:44775
I20260925 07:21:10.003857 12015 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:10.004070 12015 heartbeater.cc:511] Master 127.4.47.62:44775 requested a full tablet report, sending...
I20260925 07:21:10.004699 11811 ts_manager.cc:194] Registered new tserver with Master: d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583)
I20260925 07:21:10.005827 11811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:36513
I20260925 07:21:10.009286 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 11884
I20260925 07:21:10.009406 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:10.009650 4284 external_mini_cluster.cc:1428] Setting key 6316036bf2fa3484e89ee425a8bdeb45
I20260925 07:21:10.011535 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.005004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.1:54517)
0925 07:21:10.005144 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.005156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.005215 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.005332 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.005333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.005562 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.005641 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.005643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.006471 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.006472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.007296 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.007298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.007629 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.007629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.007681 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.008241 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.008246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.008597 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.010345 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.010346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.010346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.010557 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.010798 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.010799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.010800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.010871 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.011209 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.011211 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.011258 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.011329 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.011416 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260925 07:21:10.011610 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.005066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:54517 (local address 127.4.47.62:44775)
0925 07:21:10.005105 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.005106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.005213 (+ 107us) server_negotiation.cc:407] Connection header received
0925 07:21:10.005273 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.005274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.005281 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.005326 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.005737 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.006374 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.007372 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.007561 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.007617 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.008672 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.008673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.008673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.008678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.010260 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.010625 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.010626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.010627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.010692 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.010979 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.010980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.010980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.011147 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.011218 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.011367 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.011470 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
I20260925 07:21:10.037846 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:44775
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:10.119966 12024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:10.120215 12024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:10.120254 12024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:10.121425 12024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:10.121495 12024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:10.121528 12024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:21:10.121568 12024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:10.123049 12024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:10.123353 12024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:10.123581 12024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:10.128576 12032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:10.128569 12030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:10.128650 12024 server_base.cc:1034] running on GCE node
W20260925 07:21:10.128569 12029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:10.128958 12024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:10.129206 12024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:10.130389 12024 hybrid_clock.cc:648] HybridClock initialized: now 1790320870130375 us; error 27 us; skew 500 ppm
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:10.132964 12024 init.cc:377] Logged in from keytab as oryx/127.4.47.2@KRBTEST.COM (short username oryx)
I20260925 07:21:10.133442 12024 webserver.cc:460] Webserver started at http://127.4.47.2:35991/ using document root <none> and password file <none>
I20260925 07:21:10.133632 12024 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:10.133675 12024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:10.133764 12024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:10.134753 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "2f6f703b82d74f7386ca38a98137a397"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "4398775ef09dbbce9b3da686056f64e7"
server_key_iv: "21550de275bef768c1ac24cdecdddf6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.135105 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "2f6f703b82d74f7386ca38a98137a397"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "4398775ef09dbbce9b3da686056f64e7"
server_key_iv: "21550de275bef768c1ac24cdecdddf6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.136371 12024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:10.137135 12039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.137306 12024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:10.137384 12024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "2f6f703b82d74f7386ca38a98137a397"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "4398775ef09dbbce9b3da686056f64e7"
server_key_iv: "21550de275bef768c1ac24cdecdddf6d"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.137437 12024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:10.149168 12024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:10.150910 12024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:10.151022 12024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:10.151618 12024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:10.151659 12024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.151681 12024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:10.151695 12024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.159118 12024 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:40875
I20260925 07:21:10.159230 12154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:40875 every 8 connection(s)
I20260925 07:21:10.159514 12024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:10.162655 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.152060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55989 (local address 127.4.47.62:44775)
0925 07:21:10.152128 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.152131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.152387 (+ 256us) server_negotiation.cc:407] Connection header received
0925 07:21:10.153138 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.153139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.153151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.153200 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.154128 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.154975 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.156734 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.156955 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.157013 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.159666 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.159668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.159668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.159675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.161049 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.161555 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.161557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.161557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.161624 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.161887 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.161888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.161888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.162065 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.162132 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.162483 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.162564 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:21:10.162693 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12024
I20260925 07:21:10.162819 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:10.162838 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.152038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.2:55989)
0925 07:21:10.152231 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.152287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.153015 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.153295 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.153299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.153715 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.154020 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.154028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.155112 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.155113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.156599 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.156603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.157026 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.157027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.157186 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.157825 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.157836 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.159574 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.161121 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.161126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.161127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.161450 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.161691 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.161692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.161692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.161818 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.162123 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.162129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.162319 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.162447 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.162614 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260925 07:21:10.163017 4284 external_mini_cluster.cc:1428] Setting key 69b25d74dab791e4b1178cac2f454ecd
I20260925 07:21:10.163336 12155 heartbeater.cc:347] Connected to a master server at 127.4.47.62:44775
I20260925 07:21:10.163416 12155 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:10.163597 12155 heartbeater.cc:511] Master 127.4.47.62:44775 requested a full tablet report, sending...
I20260925 07:21:10.163973 11811 ts_manager.cc:194] Registered new tserver with Master: 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875)
I20260925 07:21:10.164455 11811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:55989
I20260925 07:21:10.171165 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.164428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.2:48221)
0925 07:21:10.164605 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.164616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.164677 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.164827 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.164828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.165044 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.165108 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.165110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.166059 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.166061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.167008 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.167011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.167499 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.167499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.167548 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.167995 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.168000 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.168351 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.170125 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.170126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.170126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.170278 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.170492 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.170492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.170493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.170542 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.170877 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.170878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.170913 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.170959 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.171038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260925 07:21:10.171196 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.164496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:48221 (local address 127.4.47.62:44775)
0925 07:21:10.164536 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.164538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.164665 (+ 127us) server_negotiation.cc:407] Connection header received
0925 07:21:10.164730 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.164731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.164741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.164788 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.165198 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.165941 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.167124 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.167420 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.167495 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.168439 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.168440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.168440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.168449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.170045 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.170350 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.170351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.170352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.170411 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.170604 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.170604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.170605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.170807 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.170875 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.170985 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.171097 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
I20260925 07:21:10.190163 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:44775
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:10.271154 12164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:10.271380 12164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:10.271426 12164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:10.272603 12164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:10.272668 12164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:10.272701 12164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:21:10.272742 12164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:10.274200 12164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.47.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:10.274464 12164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:10.274686 12164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:10.279536 12169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:10.279646 12170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:10.279697 12172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:10.279786 12164 server_base.cc:1034] running on GCE node
I20260925 07:21:10.279992 12164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:10.280236 12164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:10.281394 12164 hybrid_clock.cc:648] HybridClock initialized: now 1790320870281378 us; error 28 us; skew 500 ppm
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:10.283879 12164 init.cc:377] Logged in from keytab as oryx/127.4.47.3@KRBTEST.COM (short username oryx)
I20260925 07:21:10.284343 12164 webserver.cc:460] Webserver started at http://127.4.47.3:41259/ using document root <none> and password file <none>
I20260925 07:21:10.284584 12164 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:10.284651 12164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:10.284798 12164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:10.285846 12164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "02bfad965606499ba6f13ff684a87f19"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "db37b0909df1d704f605aca78a8a8932"
server_key_iv: "c9c2d8cd139a7dfaa36743558bebcf68"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.286207 12164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "02bfad965606499ba6f13ff684a87f19"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "db37b0909df1d704f605aca78a8a8932"
server_key_iv: "c9c2d8cd139a7dfaa36743558bebcf68"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.287576 12164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:10.288269 12179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.288551 12164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:10.288645 12164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "02bfad965606499ba6f13ff684a87f19"
format_stamp: "Formatted at 2026-09-25 07:21:10 on dist-test-slave-zw07"
server_key: "db37b0909df1d704f605aca78a8a8932"
server_key_iv: "c9c2d8cd139a7dfaa36743558bebcf68"
server_key_version: "encryptionkey@0"
I20260925 07:21:10.288713 12164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:10.301743 12164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:10.303773 12164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:10.303942 12164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:10.304551 12164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:10.304608 12164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.304651 12164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:10.304684 12164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.312076 12164 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:37463
I20260925 07:21:10.312147 12294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:37463 every 8 connection(s)
I20260925 07:21:10.312502 12164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:10.314806 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12164
I20260925 07:21:10.314942 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:10.315203 4284 external_mini_cluster.cc:1428] Setting key f11d9abab7dbfd2edc2f868da0a0a318
I20260925 07:21:10.315698 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.305151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:60353 (local address 127.4.47.62:44775)
0925 07:21:10.305227 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.305230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.305503 (+ 273us) server_negotiation.cc:407] Connection header received
0925 07:21:10.306354 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.306356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.306375 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.306457 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.307477 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.308318 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.309373 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.309714 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.309795 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.312324 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.312326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.312327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.312335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.314031 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.314479 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.314480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.314481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.314539 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.314846 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.314847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.314847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.315025 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.315156 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.315527 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.315610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:21:10.315878 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.305129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.3:60353)
0925 07:21:10.305335 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.305385 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.306124 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.306551 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.306557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.306982 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.307312 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.307320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.308418 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.308419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.309261 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.309264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.309821 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.309822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.309954 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.310553 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.310561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.312226 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.314137 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.314144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.314147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.314383 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.314614 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.314615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.314615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.314759 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.315150 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.315157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.315357 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.315488 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.315651 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20260925 07:21:10.316329 12295 heartbeater.cc:347] Connected to a master server at 127.4.47.62:44775
I20260925 07:21:10.316414 12295 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:10.316612 12295 heartbeater.cc:511] Master 127.4.47.62:44775 requested a full tablet report, sending...
I20260925 07:21:10.317044 11811 ts_manager.cc:194] Registered new tserver with Master: 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463)
I20260925 07:21:10.317565 11811 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:60353
I20260925 07:21:10.322978 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.317434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:50569 (local address 127.4.47.62:44775)
0925 07:21:10.317471 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.317472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.317544 (+ 72us) server_negotiation.cc:407] Connection header received
0925 07:21:10.317604 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.317604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.317612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.317686 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.318048 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.318521 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.319290 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.319490 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.319570 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.320496 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.320497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.320497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.320504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.321824 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.322176 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.322177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.322177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.322240 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.322455 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.322455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.322456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.322567 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.322640 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.322816 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.322889 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:21:10.322995 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.317367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.4.47.3:50569)
0925 07:21:10.317488 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.317498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.317550 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.317711 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.317712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.317906 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.317966 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.317968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.318613 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.318614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.319207 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.319209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.319571 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.319572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.319637 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.320083 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.320088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.320387 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.321927 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.321928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.321928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.322095 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.322323 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.322324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.322325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.322382 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.322645 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.322646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.322691 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.322788 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.322891 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260925 07:21:10.327047 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:21:10.348469 11862 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:10.342836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.4.47.62:44775)
0925 07:21:10.342901 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.342903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.342910 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:10.342939 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.342939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.342948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.343024 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.343753 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.344249 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.345005 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.345199 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.345284 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.348371 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51278: BlockingRecv error: recv got EOF from 127.0.0.1:51278 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.355233 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.349225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.4.47.62:44775)
0925 07:21:10.349277 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.349278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.349372 (+ 94us) server_negotiation.cc:407] Connection header received
0925 07:21:10.349447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.349447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.349455 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.349502 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.349904 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.350354 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.350985 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.351201 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.351250 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.353178 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.353179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.353179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.353185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.354320 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.354625 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.354625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.354626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.354675 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.354804 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.354804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.354804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.354896 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.354966 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.355051 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.355167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:21:10.356703 11811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:10.358278 11811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:10.364804 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.4.47.2:40875)
0925 07:21:10.360843 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.360849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.360958 (+ 109us) server_negotiation.cc:407] Connection header received
0925 07:21:10.361009 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.361012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.361029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.361113 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.361323 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362198 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.363625 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364362 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364485 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.364569 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.364672 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.364707 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260925 07:21:10.364791 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40875 (local address 127.0.0.1:45750)
0925 07:21:10.360868 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.360880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.360949 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.361137 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.361140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.361142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.361218 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.361220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362277 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.362278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.363544 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.363546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364486 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.364563 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.364578 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.364657 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.364707 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":84,"threads_started":1}
I20260925 07:21:10.365216 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.4.47.3:37463)
0925 07:21:10.361130 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.361134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.361144 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:10.361181 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.361183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.361198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.361288 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.361429 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362575 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.363787 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364571 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364641 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.364714 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.365078 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.365127 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":68,"threads_started":1}
I20260925 07:21:10.365214 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:46583 (local address 127.0.0.1:60226)
0925 07:21:10.361373 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.361384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.361444 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.361648 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.361649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.361651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.361685 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.361687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362843 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.362844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.364003 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.364006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364991 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.365053 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.365057 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.365116 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.365152 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":66,"threads_started":1}
I20260925 07:21:10.365288 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.4.47.1:46583)
0925 07:21:10.361116 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.361121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.361501 (+ 380us) server_negotiation.cc:407] Connection header received
0925 07:21:10.361543 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.361545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.361559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.361631 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.361767 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362766 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364098 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364804 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364868 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.364937 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.365143 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.365183 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260925 07:21:10.365007 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.360532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:37463 (local address 127.0.0.1:52948)
0925 07:21:10.360714 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.360756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.360823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.361286 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.361286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.361288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.361344 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.361346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.362664 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.362665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.363708 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.363709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.364748 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.364749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.364829 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.364833 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.364880 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.364932 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260925 07:21:10.365895 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 (DEFAULT_TABLE table=test-table [id=49c62be033264e69a403f4e17d4eb050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.365895 12089 tablet_service.cc:1467] Processing CreateTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 (DEFAULT_TABLE table=test-table [id=49c62be033264e69a403f4e17d4eb050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.366253 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81dc10bf7c2e471bae3e9654c20f2de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.366253 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81dc10bf7c2e471bae3e9654c20f2de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.366479 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 (DEFAULT_TABLE table=test-table [id=49c62be033264e69a403f4e17d4eb050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.366816 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81dc10bf7c2e471bae3e9654c20f2de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.369719 12323 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Bootstrap starting.
I20260925 07:21:10.370244 12325 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Bootstrap starting.
I20260925 07:21:10.370314 12324 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Bootstrap starting.
I20260925 07:21:10.370441 12323 tablet_bootstrap.cc:654] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.370783 12323 log.cc:824] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:10.370831 12325 tablet_bootstrap.cc:654] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.370916 12324 tablet_bootstrap.cc:654] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.371179 12325 log.cc:824] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:10.371255 12324 log.cc:824] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:10.371696 12323 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: No bootstrap required, opened a new log
I20260925 07:21:10.371769 12323 ts_tablet_manager.cc:1397] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:21:10.371858 12325 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: No bootstrap required, opened a new log
I20260925 07:21:10.371919 12325 ts_tablet_manager.cc:1397] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.372195 12324 tablet_bootstrap.cc:492] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: No bootstrap required, opened a new log
I20260925 07:21:10.372262 12324 ts_tablet_manager.cc:1397] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.373354 12325 raft_consensus.cc:348] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.373518 12325 raft_consensus.cc:374] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.373564 12325 raft_consensus.cc:729] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.373669 12325 consensus_queue.cc:260] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.373778 12323 raft_consensus.cc:348] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.373944 12323 raft_consensus.cc:374] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.373951 12325 ts_tablet_manager.cc:1428] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:21:10.373974 12155 heartbeater.cc:503] Master 127.4.47.62:44775 was elected leader, sending a full tablet report...
I20260925 07:21:10.373991 12323 raft_consensus.cc:729] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.374100 12323 consensus_queue.cc:260] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.374338 12323 ts_tablet_manager.cc:1428] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:21:10.374480 12324 raft_consensus.cc:348] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.374598 12324 raft_consensus.cc:374] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.374619 12324 raft_consensus.cc:729] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d347df3a1f4e10800d419fe88fbcf7, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.374678 12295 heartbeater.cc:503] Master 127.4.47.62:44775 was elected leader, sending a full tablet report...
I20260925 07:21:10.374760 12324 consensus_queue.cc:260] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.375039 12324 ts_tablet_manager.cc:1428] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:21:10.375267 12015 heartbeater.cc:503] Master 127.4.47.62:44775 was elected leader, sending a full tablet report...
I20260925 07:21:10.402334 12329 raft_consensus.cc:482] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:10.402449 12329 raft_consensus.cc:504] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.402843 12329 leader_election.cc:288] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463), d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583)
I20260925 07:21:10.406782 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.403148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:48593 (local address 127.4.47.3:37463)
0925 07:21:10.403193 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.403195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.403309 (+ 114us) server_negotiation.cc:407] Connection header received
0925 07:21:10.403360 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.403361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.403372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.403449 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.403587 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.404349 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.405824 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.406379 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.406458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.406467 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.406665 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.406712 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:21:10.406772 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.403031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:37463 (local address 127.4.47.2:48593)
0925 07:21:10.403219 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.403228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.403298 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.403454 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.403455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.403457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.403522 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.403523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.404415 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.404416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.405736 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.405739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.406469 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.406470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.406558 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.406565 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.406625 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.406665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":78,"thread_start_us":124,"threads_started":1}
I20260925 07:21:10.406782 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.403099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:43099 (local address 127.4.47.1:46583)
0925 07:21:10.403148 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.403150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.403262 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:21:10.403296 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.403297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.403312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.403365 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.403586 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.404321 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.405824 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.406379 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.406457 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.406467 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.406665 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.406712 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:10.406781 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.403031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:46583 (local address 127.4.47.2:43099)
0925 07:21:10.403151 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.403162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.403215 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.403389 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.403390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.403392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.403493 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.403496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.404431 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.404432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.405736 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.405739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.406469 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.406470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.406558 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.406564 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.406625 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.406665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":64}
I20260925 07:21:10.407159 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81dc10bf7c2e471bae3e9654c20f2de5" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" is_pre_election: true
I20260925 07:21:10.407161 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81dc10bf7c2e471bae3e9654c20f2de5" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19" is_pre_election: true
I20260925 07:21:10.407318 11969 raft_consensus.cc:2454] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.407317 12249 raft_consensus.cc:2454] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.407541 12040 leader_election.cc:302] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bfad965606499ba6f13ff684a87f19, 2f6f703b82d74f7386ca38a98137a397; no voters:
I20260925 07:21:10.407716 12329 raft_consensus.cc:2781] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:10.407753 12329 raft_consensus.cc:482] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:10.407769 12329 raft_consensus.cc:3037] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.408672 12329 raft_consensus.cc:504] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.408814 12329 leader_election.cc:288] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Requested vote from peers 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463), d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583)
I20260925 07:21:10.408973 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81dc10bf7c2e471bae3e9654c20f2de5" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
I20260925 07:21:10.408973 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81dc10bf7c2e471bae3e9654c20f2de5" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19"
I20260925 07:21:10.409049 11969 raft_consensus.cc:3037] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.409049 12249 raft_consensus.cc:3037] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.409926 12249 raft_consensus.cc:2454] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
I20260925 07:21:10.409926 11969 raft_consensus.cc:2454] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
W20260925 07:21:10.410091 12156 tablet.cc:2369] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:21:10.410094 12043 leader_election.cc:302] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6f703b82d74f7386ca38a98137a397, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.410234 12329 raft_consensus.cc:2781] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:10.410382 12329 raft_consensus.cc:686] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Running, Role: LEADER
I20260925 07:21:10.410488 12329 consensus_queue.cc:237] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } }
I20260925 07:21:10.411320 11812 catalog_manager.cc:5495] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } health_report { overall_health: UNKNOWN } } }
I20260925 07:21:10.421540 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.419406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.4.47.2:40875)
0925 07:21:10.419453 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.419454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.419469 (+ 15us) server_negotiation.cc:407] Connection header received
0925 07:21:10.419533 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.419534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.419550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.419607 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.419716 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.420163 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.420855 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.421032 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.421082 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.421214 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.421316 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.421374 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.421416 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.421432 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:21:10.423321 11969 raft_consensus.cc:1261] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.423321 12249 raft_consensus.cc:1261] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.423676 12333 consensus_queue.cc:1035] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.423810 12329 consensus_queue.cc:1035] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.426597 12338 mvcc.cc:205] Tried to move back new op lower bound from 7333154285250895872 to 7333154285201895424. Current Snapshot: MvccSnapshot[applied={T|T < 7333154285250895872}]
I20260925 07:21:10.428354 11812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.428447 11812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.429176 11812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Sending DeleteTablet for 3 replicas of tablet 81dc10bf7c2e471bae3e9654c20f2de5
I20260925 07:21:10.429451 12229 tablet_service.cc:1514] Processing DeleteTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 07:21:10.429620 12089 tablet_service.cc:1514] Processing DeleteTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:45750
I20260925 07:21:10.429673 12346 tablet_replica.cc:330] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: stopping tablet replica
I20260925 07:21:10.429812 11949 tablet_service.cc:1514] Processing DeleteTablet for tablet 81dc10bf7c2e471bae3e9654c20f2de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260925 07:21:10.429926 12346 raft_consensus.cc:2229] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.429865 11812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:10.430016 11812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:10.430023 12347 tablet_replica.cc:330] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: stopping tablet replica
I20260925 07:21:10.430102 12347 raft_consensus.cc:2229] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:10.430192 12347 raft_consensus.cc:2258] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.430302 12346 raft_consensus.cc:2258] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.430686 12348 tablet_replica.cc:330] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: stopping tablet replica
I20260925 07:21:10.430781 12348 raft_consensus.cc:2229] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.431391 12346 ts_tablet_manager.cc:1905] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.431439 11949 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab (DEFAULT_TABLE table=test-table [id=3608000be2e04f959228e30cd6e7b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.431536 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a6b2036934f9d953c3d2fe44fd2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.432967 12324 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Bootstrap starting.
I20260925 07:21:10.433355 12229 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab (DEFAULT_TABLE table=test-table [id=3608000be2e04f959228e30cd6e7b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.433405 12324 tablet_bootstrap.cc:654] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.433001 12089 tablet_service.cc:1467] Processing CreateTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab (DEFAULT_TABLE table=test-table [id=3608000be2e04f959228e30cd6e7b448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.433513 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a6b2036934f9d953c3d2fe44fd2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.433518 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1a6b2036934f9d953c3d2fe44fd2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.434705 12323 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Bootstrap starting.
I20260925 07:21:10.434805 12346 ts_tablet_manager.cc:1918] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.434871 12346 log.cc:1196] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/81dc10bf7c2e471bae3e9654c20f2de5
I20260925 07:21:10.434888 12325 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Bootstrap starting.
I20260925 07:21:10.435154 12323 tablet_bootstrap.cc:654] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.435195 12346 ts_tablet_manager.cc:1939] T 81dc10bf7c2e471bae3e9654c20f2de5 P 02bfad965606499ba6f13ff684a87f19: Deleting consensus metadata
I20260925 07:21:10.435322 12325 tablet_bootstrap.cc:654] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.435595 11796 catalog_manager.cc:4841] TS 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463): tablet 81dc10bf7c2e471bae3e9654c20f2de5 (table test-table [id=49c62be033264e69a403f4e17d4eb050]) successfully deleted
I20260925 07:21:10.435845 12348 raft_consensus.cc:2258] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.436210 12347 ts_tablet_manager.cc:1905] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.436261 12348 ts_tablet_manager.cc:1905] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.436278 12325 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: No bootstrap required, opened a new log
I20260925 07:21:10.436448 12325 ts_tablet_manager.cc:1397] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.436559 12323 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: No bootstrap required, opened a new log
I20260925 07:21:10.436599 12323 ts_tablet_manager.cc:1397] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.436604 12325 raft_consensus.cc:348] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.436668 12325 raft_consensus.cc:374] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.436688 12325 raft_consensus.cc:729] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.436722 12323 raft_consensus.cc:348] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.436744 12325 consensus_queue.cc:260] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.436780 12323 raft_consensus.cc:374] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.436800 12323 raft_consensus.cc:729] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.436822 12325 ts_tablet_manager.cc:1428] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.436851 12323 consensus_queue.cc:260] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.436957 12323 ts_tablet_manager.cc:1428] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.437896 12348 ts_tablet_manager.cc:1918] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.437976 12348 log.cc:1196] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/81dc10bf7c2e471bae3e9654c20f2de5
I20260925 07:21:10.438261 12348 ts_tablet_manager.cc:1939] T 81dc10bf7c2e471bae3e9654c20f2de5 P d9d347df3a1f4e10800d419fe88fbcf7: Deleting consensus metadata
I20260925 07:21:10.438668 11799 catalog_manager.cc:4841] TS d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583): tablet 81dc10bf7c2e471bae3e9654c20f2de5 (table test-table [id=49c62be033264e69a403f4e17d4eb050]) successfully deleted
I20260925 07:21:10.439049 12347 ts_tablet_manager.cc:1918] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.439145 12347 log.cc:1196] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/81dc10bf7c2e471bae3e9654c20f2de5
I20260925 07:21:10.439395 12324 tablet_bootstrap.cc:492] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: No bootstrap required, opened a new log
I20260925 07:21:10.439399 12347 ts_tablet_manager.cc:1939] T 81dc10bf7c2e471bae3e9654c20f2de5 P 2f6f703b82d74f7386ca38a98137a397: Deleting consensus metadata
I20260925 07:21:10.439471 12324 ts_tablet_manager.cc:1397] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 07:21:10.439626 12324 raft_consensus.cc:348] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.439731 12324 raft_consensus.cc:374] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.439760 12324 raft_consensus.cc:729] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d347df3a1f4e10800d419fe88fbcf7, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.439807 12324 consensus_queue.cc:260] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.439831 11799 catalog_manager.cc:4841] TS 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875): tablet 81dc10bf7c2e471bae3e9654c20f2de5 (table test-table [id=49c62be033264e69a403f4e17d4eb050]) successfully deleted
I20260925 07:21:10.439911 12324 ts_tablet_manager.cc:1428] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:10.499503 12330 raft_consensus.cc:482] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:10.499626 12330 raft_consensus.cc:504] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
W20260925 07:21:10.499857 12016 tablet.cc:2369] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:21:10.499971 12330 leader_election.cc:288] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875)
I20260925 07:21:10.503829 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.500441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:58113 (local address 127.4.47.2:40875)
0925 07:21:10.500487 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.500489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.500521 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:21:10.500604 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.500605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.500613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.500664 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.500832 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.501302 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.502824 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.503403 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.503481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.503493 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.503712 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.503755 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:21:10.503839 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.500268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40875 (local address 127.4.47.3:58113)
0925 07:21:10.500462 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.500474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.500528 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.500666 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.500667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.500669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.500758 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.500760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.501412 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.501413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.502716 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.502719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.503515 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.503517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.503586 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.503591 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.503684 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.503729 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":44,"thread_start_us":66,"threads_started":1}
I20260925 07:21:10.504052 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.500161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:46583 (local address 127.4.47.3:42507)
0925 07:21:10.500206 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.500219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.500268 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.500468 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.500470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.500472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:10.500544 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.500547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.501460 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.501461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.502716 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.502719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.503794 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.503794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.503871 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.503873 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.503926 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.503973 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":70}
I20260925 07:21:10.504283 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a6b2036934f9d953c3d2fe44fd2ab" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6f703b82d74f7386ca38a98137a397" is_pre_election: true
I20260925 07:21:10.504436 12109 raft_consensus.cc:2454] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 0.
I20260925 07:21:10.504593 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.500238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:42507 (local address 127.4.47.1:46583)
0925 07:21:10.500282 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.500285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.500295 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:10.500329 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.500330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.500341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.500414 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:10.500627 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.501386 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.503151 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.503697 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.503753 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.503762 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.504491 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.504532 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:21:10.504662 12183 leader_election.cc:302] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bfad965606499ba6f13ff684a87f19, 2f6f703b82d74f7386ca38a98137a397; no voters:
I20260925 07:21:10.504799 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a6b2036934f9d953c3d2fe44fd2ab" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" is_pre_election: true
I20260925 07:21:10.504885 11969 raft_consensus.cc:2454] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 0.
I20260925 07:21:10.505038 12330 raft_consensus.cc:2781] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:10.505086 12330 raft_consensus.cc:482] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:10.505110 12330 raft_consensus.cc:3037] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.506031 12330 raft_consensus.cc:504] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.506167 12330 leader_election.cc:288] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 election: Requested vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875)
I20260925 07:21:10.506382 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a6b2036934f9d953c3d2fe44fd2ab" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
I20260925 07:21:10.506397 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1a6b2036934f9d953c3d2fe44fd2ab" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6f703b82d74f7386ca38a98137a397"
I20260925 07:21:10.506460 11969 raft_consensus.cc:3037] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.506469 12109 raft_consensus.cc:3037] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.507213 12109 raft_consensus.cc:2454] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 1.
I20260925 07:21:10.507238 11969 raft_consensus.cc:2454] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 1.
I20260925 07:21:10.507376 12183 leader_election.cc:302] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bfad965606499ba6f13ff684a87f19, 2f6f703b82d74f7386ca38a98137a397; no voters:
I20260925 07:21:10.507480 12330 raft_consensus.cc:2781] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:10.507635 12330 raft_consensus.cc:686] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 1 LEADER]: Becoming Leader. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Running, Role: LEADER
I20260925 07:21:10.507709 12330 consensus_queue.cc:237] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.508477 11812 catalog_manager.cc:5495] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bfad965606499ba6f13ff684a87f19 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "02bfad965606499ba6f13ff684a87f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } health_report { overall_health: HEALTHY } } }
W20260925 07:21:10.563161 12296 tablet.cc:2369] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320869, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.62@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.566886 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.559848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:44775 (local address 127.0.0.1:51290)
0925 07:21:10.560059 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.560068 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.560159 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.560310 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.560311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.560659 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:10.560706 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.560709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.561405 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.561406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.561926 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.561928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.562312 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.562313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.562369 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.562898 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:10.562903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:10.564414 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:10.565822 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.565830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.565832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.566027 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.566231 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:10.566232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:10.566232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:10.566290 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:10.566548 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:10.566554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:10.566603 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.566694 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.566815 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":116,"threads_started":1}
I20260925 07:21:10.566968 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.559922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.4.47.62:44775)
0925 07:21:10.560004 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.560006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.560138 (+ 132us) server_negotiation.cc:407] Connection header received
0925 07:21:10.560224 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.560225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.560241 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.560306 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.560806 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.561312 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.562036 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.562213 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.562299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.564519 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.564521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.564521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.564527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.565726 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.566112 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.566113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.566114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.566168 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.566357 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.566358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.566358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.566472 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.566543 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.566753 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.566890 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":34}
I20260925 07:21:10.568761 11812 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:51290:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:21:10.568969 11812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:10.570341 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad37afb404130aa3a2d314039e950 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937e866630d0441b9f876dddee9b088b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:21:10.570338 12089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad37afb404130aa3a2d314039e950 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937e866630d0441b9f876dddee9b088b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:21:10.570343 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecad37afb404130aa3a2d314039e950 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937e866630d0441b9f876dddee9b088b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:21:10.570505 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad37afb404130aa3a2d314039e950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.570505 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad37afb404130aa3a2d314039e950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.570518 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecad37afb404130aa3a2d314039e950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.571712 12324 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: Bootstrap starting.
I20260925 07:21:10.571990 12323 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: Bootstrap starting.
I20260925 07:21:10.572005 12325 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: Bootstrap starting.
I20260925 07:21:10.572152 12324 tablet_bootstrap.cc:654] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.572309 12325 tablet_bootstrap.cc:654] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.572333 12323 tablet_bootstrap.cc:654] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.572846 12325 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: No bootstrap required, opened a new log
I20260925 07:21:10.572846 12324 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: No bootstrap required, opened a new log
I20260925 07:21:10.572901 12324 ts_tablet_manager.cc:1397] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:10.572901 12325 ts_tablet_manager.cc:1397] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:10.573020 12325 raft_consensus.cc:348] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573048 12324 raft_consensus.cc:348] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573096 12325 raft_consensus.cc:374] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.573108 12324 raft_consensus.cc:374] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.573118 12325 raft_consensus.cc:729] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.573110 12323 tablet_bootstrap.cc:492] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: No bootstrap required, opened a new log
I20260925 07:21:10.573134 12324 raft_consensus.cc:729] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d347df3a1f4e10800d419fe88fbcf7, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.573158 12323 ts_tablet_manager.cc:1397] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:10.573165 12325 consensus_queue.cc:260] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573180 12324 consensus_queue.cc:260] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573256 12325 ts_tablet_manager.cc:1428] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.573271 12324 ts_tablet_manager.cc:1428] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:10.573297 12323 raft_consensus.cc:348] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573370 12323 raft_consensus.cc:374] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.573397 12323 raft_consensus.cc:729] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.573375 12329 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.573441 12329 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.573457 12323 consensus_queue.cc:260] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.573557 12323 ts_tablet_manager.cc:1428] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:10.573583 12331 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.573683 12331 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.573675 12330 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.573752 12330 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.642863 12329 raft_consensus.cc:482] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:10.642961 12329 raft_consensus.cc:504] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.643175 12329 leader_election.cc:288] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463)
I20260925 07:21:10.643421 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad37afb404130aa3a2d314039e950" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19" is_pre_election: true
I20260925 07:21:10.643429 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad37afb404130aa3a2d314039e950" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" is_pre_election: true
I20260925 07:21:10.643527 11969 raft_consensus.cc:2454] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.643527 12249 raft_consensus.cc:2454] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.643664 12043 leader_election.cc:302] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6f703b82d74f7386ca38a98137a397, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.643769 12329 raft_consensus.cc:2781] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:10.643808 12329 raft_consensus.cc:482] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:10.643829 12329 raft_consensus.cc:3037] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.644589 12329 raft_consensus.cc:504] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.644726 12329 leader_election.cc:288] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Requested vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463)
I20260925 07:21:10.644848 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad37afb404130aa3a2d314039e950" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
I20260925 07:21:10.644936 11969 raft_consensus.cc:3037] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.644909 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecad37afb404130aa3a2d314039e950" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19"
I20260925 07:21:10.644975 12249 raft_consensus.cc:3037] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.645751 11969 raft_consensus.cc:2454] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
I20260925 07:21:10.645925 12043 leader_election.cc:302] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6f703b82d74f7386ca38a98137a397, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.646020 12329 raft_consensus.cc:2781] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:10.646075 12329 raft_consensus.cc:686] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Running, Role: LEADER
I20260925 07:21:10.646111 12249 raft_consensus.cc:2454] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
I20260925 07:21:10.646132 12329 consensus_queue.cc:237] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.646384 12343 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: TxnStatusTablet state changed. Reason: New leader 2f6f703b82d74f7386ca38a98137a397. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.646487 12343 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:21:10.646754 12365 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:21:10.646878 12365 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:21:10.646910 11811 catalog_manager.cc:5495] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260925 07:21:10.667644 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.665015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.4.47.2:40875)
0925 07:21:10.665063 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.665065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.665113 (+ 48us) server_negotiation.cc:407] Connection header received
0925 07:21:10.665194 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.665195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.665208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.665268 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.665411 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.665879 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.666962 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.667208 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.667260 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.667447 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.667472 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.667509 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.667569 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.667584 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:21:10.667689 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.664964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40875 (local address 127.0.0.1:45776)
0925 07:21:10.665062 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.665073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.665125 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.665270 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.665271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.665272 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:21:10.665315 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.665317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.665967 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.665969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.666881 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.666884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.667275 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.667276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.667345 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.667364 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.667510 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.667515 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.667565 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.667608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 07:21:10.668819 11969 raft_consensus.cc:1261] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.668831 12249 raft_consensus.cc:1261] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.669076 12343 consensus_queue.cc:1035] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.669188 12329 consensus_queue.cc:1035] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.669718 12330 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: TxnStatusTablet state changed. Reason: New leader 2f6f703b82d74f7386ca38a98137a397. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.669739 12331 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: TxnStatusTablet state changed. Reason: New leader 2f6f703b82d74f7386ca38a98137a397. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.669814 12330 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.669826 12331 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.670395 12329 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.670485 12329 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:21:10.670558 12330 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.670614 12337 mvcc.cc:205] Tried to move back new op lower bound from 7333154286257823744 to 7333154286167035904. Current Snapshot: MvccSnapshot[applied={T|T < 7333154286257823744}]
I20260925 07:21:10.670632 12330 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 02bfad965606499ba6f13ff684a87f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.671448 12329 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.671528 12329 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P 2f6f703b82d74f7386ca38a98137a397: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:21:10.671667 12331 tablet_replica.cc:439] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } } }
I20260925 07:21:10.671746 12331 tablet_replica.cc:442] T 2ecad37afb404130aa3a2d314039e950 P d9d347df3a1f4e10800d419fe88fbcf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:21:10.675016 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.672179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.4.47.3:37463)
0925 07:21:10.672275 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.672277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.672284 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:10.672314 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.672315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.672333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.672404 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.672558 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.673199 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.674271 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.674460 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.674514 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.674695 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.674796 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.674845 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.674917 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.674932 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 07:21:10.679824 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.677343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:43345 (local address 127.4.47.2:40875)
0925 07:21:10.677406 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.677408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.677415 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:10.677476 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.677477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.677488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.677533 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.677677 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.678113 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.679193 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.679367 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.679415 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.679617 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.679643 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.679677 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.679753 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.679771 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:21:10.679872 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.677277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:40875 (local address 127.4.47.3:43345)
0925 07:21:10.677359 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.677371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.677416 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.677556 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.677556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.677558 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:21:10.677617 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.677619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.678179 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.678180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.679107 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.679109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.679433 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.679434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.679514 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.679533 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.679697 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.679699 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.679752 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.679796 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:21:10.682154 11969 raft_consensus.cc:1261] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Refusing update from remote peer 02bfad965606499ba6f13ff684a87f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.682241 12109 raft_consensus.cc:1261] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Refusing update from remote peer 02bfad965606499ba6f13ff684a87f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.682518 12353 consensus_queue.cc:1035] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.682664 12330 consensus_queue.cc:1035] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.684381 12336 mvcc.cc:205] Tried to move back new op lower bound from 7333154286311886848 to 7333154285600165888. Current Snapshot: MvccSnapshot[applied={T|T < 7333154286311886848}]
I20260925 07:21:10.691143 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.688159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56103 (local address 127.4.47.3:37463)
0925 07:21:10.688202 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.688204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.688463 (+ 259us) server_negotiation.cc:407] Connection header received
0925 07:21:10.688511 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.688512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.688525 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.688582 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.688764 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.689501 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.690415 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.690603 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.690686 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.690870 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.690906 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.690971 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.691031 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.691050 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:21:10.691176 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.688114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:37463 (local address 127.4.47.2:56103)
0925 07:21:10.688338 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:10.688402 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:10.688450 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:10.688627 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:10.688628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:10.688630 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:21:10.688698 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.688699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.689579 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.689580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.690300 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:10.690303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.690692 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.690693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:10.690771 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.690785 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.690985 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.690988 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:10.691030 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 07:21:10.691101 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260925 07:21:10.698503 11810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.698642 11810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.699335 11810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Sending DeleteTablet for 3 replicas of tablet bc1a6b2036934f9d953c3d2fe44fd2ab
I20260925 07:21:10.699671 11949 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260925 07:21:10.699718 12229 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 07:21:10.699792 12346 tablet_replica.cc:330] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: stopping tablet replica
I20260925 07:21:10.699815 12348 tablet_replica.cc:330] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: stopping tablet replica
I20260925 07:21:10.699836 12346 raft_consensus.cc:2229] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:10.699858 12348 raft_consensus.cc:2229] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.699927 12348 raft_consensus.cc:2258] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.699931 12346 raft_consensus.cc:2258] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.700057 12089 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1a6b2036934f9d953c3d2fe44fd2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:45750
I20260925 07:21:10.700191 12347 tablet_replica.cc:330] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: stopping tablet replica
I20260925 07:21:10.700275 12347 raft_consensus.cc:2229] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.700284 12346 ts_tablet_manager.cc:1905] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.700320 12347 raft_consensus.cc:2258] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.700644 12347 ts_tablet_manager.cc:1905] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.700726 12348 ts_tablet_manager.cc:1905] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.702001 12346 ts_tablet_manager.cc:1918] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:21:10.702057 12346 log.cc:1196] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/bc1a6b2036934f9d953c3d2fe44fd2ab
I20260925 07:21:10.702270 12346 ts_tablet_manager.cc:1939] T bc1a6b2036934f9d953c3d2fe44fd2ab P 02bfad965606499ba6f13ff684a87f19: Deleting consensus metadata
I20260925 07:21:10.702720 11796 catalog_manager.cc:4841] TS 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463): tablet bc1a6b2036934f9d953c3d2fe44fd2ab (table test-table [id=3608000be2e04f959228e30cd6e7b448]) successfully deleted
I20260925 07:21:10.704342 12347 ts_tablet_manager.cc:1918] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:21:10.704403 12347 log.cc:1196] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/bc1a6b2036934f9d953c3d2fe44fd2ab
I20260925 07:21:10.704612 12347 ts_tablet_manager.cc:1939] T bc1a6b2036934f9d953c3d2fe44fd2ab P 2f6f703b82d74f7386ca38a98137a397: Deleting consensus metadata
I20260925 07:21:10.704948 11799 catalog_manager.cc:4841] TS 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875): tablet bc1a6b2036934f9d953c3d2fe44fd2ab (table test-table [id=3608000be2e04f959228e30cd6e7b448]) successfully deleted
I20260925 07:21:10.707275 12348 ts_tablet_manager.cc:1918] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:21:10.707366 12348 log.cc:1196] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/bc1a6b2036934f9d953c3d2fe44fd2ab
I20260925 07:21:10.707571 12348 ts_tablet_manager.cc:1939] T bc1a6b2036934f9d953c3d2fe44fd2ab P d9d347df3a1f4e10800d419fe88fbcf7: Deleting consensus metadata
I20260925 07:21:10.708055 11799 catalog_manager.cc:4841] TS d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583): tablet bc1a6b2036934f9d953c3d2fe44fd2ab (table test-table [id=3608000be2e04f959228e30cd6e7b448]) successfully deleted
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:21:10.730780 11862 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:10.725532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.4.47.62:44775)
0925 07:21:10.725581 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.725583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.725684 (+ 101us) server_negotiation.cc:407] Connection header received
0925 07:21:10.725767 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.725768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.725779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.725835 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.726434 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.726923 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.727748 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.727949 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.728031 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.730719 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51304: BlockingRecv error: recv got EOF from 127.0.0.1:51304 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 07:21:10 dist-test-slave-zw07 krb5kdc[11767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM
I20260925 07:21:10.738641 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.731797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.4.47.62:44775)
0925 07:21:10.731834 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.731836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.731909 (+ 73us) server_negotiation.cc:407] Connection header received
0925 07:21:10.731976 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.731977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.731986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.732041 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:10.732479 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.732955 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.733624 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.733825 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.733913 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.736115 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:10.736117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:10.736118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:10.736124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:10.737508 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.737878 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.737879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.737879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.737940 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:10.738137 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:10.738138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:10.738138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:10.738250 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:10.738353 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.738455 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.738530 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:21:10.740279 11810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:10.740476 11810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:10.741822 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 204d3863284f448e9f438541bffa710c (DEFAULT_TABLE table=test-table [id=ec9cd05640184a6c8431ba0e891e88ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.741822 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 204d3863284f448e9f438541bffa710c (DEFAULT_TABLE table=test-table [id=ec9cd05640184a6c8431ba0e891e88ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.741954 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204d3863284f448e9f438541bffa710c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.741954 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204d3863284f448e9f438541bffa710c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.743270 12323 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Bootstrap starting.
I20260925 07:21:10.743316 12324 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Bootstrap starting.
I20260925 07:21:10.743696 12323 tablet_bootstrap.cc:654] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.743742 12324 tablet_bootstrap.cc:654] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.744138 12089 tablet_service.cc:1467] Processing CreateTablet for tablet 204d3863284f448e9f438541bffa710c (DEFAULT_TABLE table=test-table [id=ec9cd05640184a6c8431ba0e891e88ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.744259 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204d3863284f448e9f438541bffa710c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.745157 12325 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Bootstrap starting.
I20260925 07:21:10.745471 12325 tablet_bootstrap.cc:654] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.745532 12323 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: No bootstrap required, opened a new log
I20260925 07:21:10.745574 12323 ts_tablet_manager.cc:1397] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.745733 12323 raft_consensus.cc:348] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.745805 12323 raft_consensus.cc:374] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.745836 12323 raft_consensus.cc:729] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.745872 12323 consensus_queue.cc:260] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.745955 12323 ts_tablet_manager.cc:1428] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:10.746676 12325 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: No bootstrap required, opened a new log
I20260925 07:21:10.746744 12325 ts_tablet_manager.cc:1397] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:21:10.746917 12325 raft_consensus.cc:348] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.746994 12325 raft_consensus.cc:374] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.747020 12325 raft_consensus.cc:729] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.747061 12325 consensus_queue.cc:260] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.747196 12325 ts_tablet_manager.cc:1428] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:10.751482 12324 tablet_bootstrap.cc:492] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: No bootstrap required, opened a new log
I20260925 07:21:10.751545 12324 ts_tablet_manager.cc:1397] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 07:21:10.751686 12324 raft_consensus.cc:348] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.751752 12324 raft_consensus.cc:374] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.751770 12324 raft_consensus.cc:729] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d347df3a1f4e10800d419fe88fbcf7, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.751809 12324 consensus_queue.cc:260] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.751919 12324 ts_tablet_manager.cc:1428] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.935283 12329 raft_consensus.cc:482] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:10.935395 12329 raft_consensus.cc:504] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.935614 12329 leader_election.cc:288] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463)
I20260925 07:21:10.935873 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204d3863284f448e9f438541bffa710c" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19" is_pre_election: true
I20260925 07:21:10.935883 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204d3863284f448e9f438541bffa710c" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" is_pre_election: true
I20260925 07:21:10.935981 12249 raft_consensus.cc:2454] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.935985 11969 raft_consensus.cc:2454] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 0.
I20260925 07:21:10.936126 12043 leader_election.cc:302] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6f703b82d74f7386ca38a98137a397, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.936256 12329 raft_consensus.cc:2781] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:10.936311 12329 raft_consensus.cc:482] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:10.936326 12329 raft_consensus.cc:3037] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.936935 12329 raft_consensus.cc:504] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.937077 12329 leader_election.cc:288] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Requested vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463)
I20260925 07:21:10.937225 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204d3863284f448e9f438541bffa710c" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
I20260925 07:21:10.937297 11969 raft_consensus.cc:3037] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.937258 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204d3863284f448e9f438541bffa710c" candidate_uuid: "2f6f703b82d74f7386ca38a98137a397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bfad965606499ba6f13ff684a87f19"
I20260925 07:21:10.937363 12249 raft_consensus.cc:3037] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.937891 11969 raft_consensus.cc:2454] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
I20260925 07:21:10.938053 12043 leader_election.cc:302] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6f703b82d74f7386ca38a98137a397, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.938084 12249 raft_consensus.cc:2454] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6f703b82d74f7386ca38a98137a397 in term 1.
I20260925 07:21:10.938148 12329 raft_consensus.cc:2781] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:10.938206 12329 raft_consensus.cc:686] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Running, Role: LEADER
I20260925 07:21:10.938261 12329 consensus_queue.cc:237] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.938875 11810 catalog_manager.cc:5495] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "2f6f703b82d74f7386ca38a98137a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260925 07:21:10.972574 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.970476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.4.47.2:40875)
0925 07:21:10.970541 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.970543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.970551 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:10.970576 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.970577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.970587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.970630 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.970761 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.971278 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.971972 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.972146 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.972193 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.972314 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.972339 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.972375 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.972437 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.972452 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:21:10.973459 11969 raft_consensus.cc:1261] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.973462 12249 raft_consensus.cc:1261] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Refusing update from remote peer 2f6f703b82d74f7386ca38a98137a397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:10.973683 12343 consensus_queue.cc:1035] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.973814 12329 consensus_queue.cc:1035] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:10.976358 11811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.976445 11811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:10.977039 11811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Sending DeleteTablet for 3 replicas of tablet 204d3863284f448e9f438541bffa710c
I20260925 07:21:10.977268 12229 tablet_service.cc:1514] Processing DeleteTablet for tablet 204d3863284f448e9f438541bffa710c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 07:21:10.977267 11949 tablet_service.cc:1514] Processing DeleteTablet for tablet 204d3863284f448e9f438541bffa710c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260925 07:21:10.977285 12089 tablet_service.cc:1514] Processing DeleteTablet for tablet 204d3863284f448e9f438541bffa710c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:10 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:45750
I20260925 07:21:10.977363 12346 tablet_replica.cc:330] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: stopping tablet replica
I20260925 07:21:10.977401 12347 tablet_replica.cc:330] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: stopping tablet replica
I20260925 07:21:10.977432 12346 raft_consensus.cc:2229] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.977455 12347 raft_consensus.cc:2229] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:10.977481 12346 raft_consensus.cc:2258] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.977533 12347 raft_consensus.cc:2258] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.977557 12348 tablet_replica.cc:330] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: stopping tablet replica
I20260925 07:21:10.977610 12348 raft_consensus.cc:2229] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:10.977659 12348 raft_consensus.cc:2258] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:10.977849 11811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 07:21:10.977958 12348 ts_tablet_manager.cc:1905] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:21:10.977994 11811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:10.978071 12347 ts_tablet_manager.cc:1905] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.979044 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 24d92cb09549478391f517c5069d3168 (DEFAULT_TABLE table=test-table [id=d258ffa3fc334dfe9afed2720587f3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.979164 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d92cb09549478391f517c5069d3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.979277 12346 ts_tablet_manager.cc:1905] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:10.979568 12348 ts_tablet_manager.cc:1918] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.979579 12347 ts_tablet_manager.cc:1918] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.979615 12348 log.cc:1196] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/204d3863284f448e9f438541bffa710c
I20260925 07:21:10.979626 12347 log.cc:1196] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/204d3863284f448e9f438541bffa710c
I20260925 07:21:10.979828 12348 ts_tablet_manager.cc:1939] T 204d3863284f448e9f438541bffa710c P d9d347df3a1f4e10800d419fe88fbcf7: Deleting consensus metadata
I20260925 07:21:10.979844 12347 ts_tablet_manager.cc:1939] T 204d3863284f448e9f438541bffa710c P 2f6f703b82d74f7386ca38a98137a397: Deleting consensus metadata
I20260925 07:21:10.980309 11799 catalog_manager.cc:4841] TS 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875): tablet 204d3863284f448e9f438541bffa710c (table test-table [id=ec9cd05640184a6c8431ba0e891e88ab]) successfully deleted
I20260925 07:21:10.980454 12324 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: Bootstrap starting.
I20260925 07:21:10.980499 12089 tablet_service.cc:1467] Processing CreateTablet for tablet 24d92cb09549478391f517c5069d3168 (DEFAULT_TABLE table=test-table [id=d258ffa3fc334dfe9afed2720587f3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.980585 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d92cb09549478391f517c5069d3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.980748 12346 ts_tablet_manager.cc:1918] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:10.980788 12346 log.cc:1196] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/204d3863284f448e9f438541bffa710c
I20260925 07:21:10.980937 12346 ts_tablet_manager.cc:1939] T 204d3863284f448e9f438541bffa710c P 02bfad965606499ba6f13ff684a87f19: Deleting consensus metadata
I20260925 07:21:10.981073 12324 tablet_bootstrap.cc:654] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.981686 12324 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: No bootstrap required, opened a new log
I20260925 07:21:10.981726 12324 ts_tablet_manager.cc:1397] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 07:21:10.981899 12325 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Bootstrap starting.
I20260925 07:21:10.981877 12324 raft_consensus.cc:348] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.981953 12324 raft_consensus.cc:374] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.981989 12324 raft_consensus.cc:729] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d347df3a1f4e10800d419fe88fbcf7, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.981977 11799 catalog_manager.cc:4841] TS d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583): tablet 204d3863284f448e9f438541bffa710c (table test-table [id=ec9cd05640184a6c8431ba0e891e88ab]) successfully deleted
I20260925 07:21:10.982036 12324 consensus_queue.cc:260] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.982125 12324 ts_tablet_manager.cc:1428] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.982152 12229 tablet_service.cc:1467] Processing CreateTablet for tablet 24d92cb09549478391f517c5069d3168 (DEFAULT_TABLE table=test-table [id=d258ffa3fc334dfe9afed2720587f3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:10.982240 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d92cb09549478391f517c5069d3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:10.982239 12325 tablet_bootstrap.cc:654] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.982539 11796 catalog_manager.cc:4841] TS 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463): tablet 204d3863284f448e9f438541bffa710c (table test-table [id=ec9cd05640184a6c8431ba0e891e88ab]) successfully deleted
I20260925 07:21:10.982779 12325 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: No bootstrap required, opened a new log
I20260925 07:21:10.982836 12325 ts_tablet_manager.cc:1397] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:10.982962 12325 raft_consensus.cc:348] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.983037 12325 raft_consensus.cc:374] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.983057 12325 raft_consensus.cc:729] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6f703b82d74f7386ca38a98137a397, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.983129 12325 consensus_queue.cc:260] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.983227 12325 ts_tablet_manager.cc:1428] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.983448 12323 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Bootstrap starting.
I20260925 07:21:10.983767 12323 tablet_bootstrap.cc:654] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:10.984280 12323 tablet_bootstrap.cc:492] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: No bootstrap required, opened a new log
I20260925 07:21:10.984318 12323 ts_tablet_manager.cc:1397] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:10.984427 12323 raft_consensus.cc:348] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.984489 12323 raft_consensus.cc:374] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:10.984504 12323 raft_consensus.cc:729] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Initialized, Role: FOLLOWER
I20260925 07:21:10.984568 12323 consensus_queue.cc:260] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.984659 12323 ts_tablet_manager.cc:1428] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:10.988796 12353 raft_consensus.cc:482] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:10.988848 12353 raft_consensus.cc:504] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.988994 12353 leader_election.cc:288] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875)
I20260925 07:21:10.989141 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d92cb09549478391f517c5069d3168" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" is_pre_election: true
I20260925 07:21:10.989166 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d92cb09549478391f517c5069d3168" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6f703b82d74f7386ca38a98137a397" is_pre_election: true
I20260925 07:21:10.989219 11969 raft_consensus.cc:2454] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 0.
I20260925 07:21:10.989250 12109 raft_consensus.cc:2454] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 0.
I20260925 07:21:10.989377 12183 leader_election.cc:302] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bfad965606499ba6f13ff684a87f19, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.989482 12353 raft_consensus.cc:2781] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:10.989521 12353 raft_consensus.cc:482] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:10.989548 12353 raft_consensus.cc:3037] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.990264 12353 raft_consensus.cc:504] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.990381 12353 leader_election.cc:288] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 election: Requested vote from peers d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583), 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875)
I20260925 07:21:10.990545 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d92cb09549478391f517c5069d3168" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d347df3a1f4e10800d419fe88fbcf7"
I20260925 07:21:10.990540 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d92cb09549478391f517c5069d3168" candidate_uuid: "02bfad965606499ba6f13ff684a87f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6f703b82d74f7386ca38a98137a397"
I20260925 07:21:10.990604 12109 raft_consensus.cc:3037] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.990619 11969 raft_consensus.cc:3037] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:10.991261 12109 raft_consensus.cc:2454] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 1.
I20260925 07:21:10.991261 11969 raft_consensus.cc:2454] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bfad965606499ba6f13ff684a87f19 in term 1.
I20260925 07:21:10.991415 12183 leader_election.cc:302] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bfad965606499ba6f13ff684a87f19, d9d347df3a1f4e10800d419fe88fbcf7; no voters:
I20260925 07:21:10.991516 12353 raft_consensus.cc:2781] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:10.991569 12353 raft_consensus.cc:686] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 1 LEADER]: Becoming Leader. State: Replica: 02bfad965606499ba6f13ff684a87f19, State: Running, Role: LEADER
I20260925 07:21:10.991616 12353 consensus_queue.cc:237] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } }
I20260925 07:21:10.992030 11811 catalog_manager.cc:5495] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bfad965606499ba6f13ff684a87f19 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "02bfad965606499ba6f13ff684a87f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bfad965606499ba6f13ff684a87f19" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37463 } health_report { overall_health: HEALTHY } } }
I20260925 07:21:10.998499 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:10.996495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.4.47.3:37463)
0925 07:21:10.996555 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:10.996556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:10.996585 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:21:10.996635 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:10.996635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:10.996643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:10.996687 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:10.996760 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.997296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.997963 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:10.998142 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:10.998188 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:10.998294 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:10.998318 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:10.998355 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:10.998399 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 07:21:10.998432 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:21:11.000336 12109 raft_consensus.cc:1261] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Refusing update from remote peer 02bfad965606499ba6f13ff684a87f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:11.000325 11969 raft_consensus.cc:1261] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Refusing update from remote peer 02bfad965606499ba6f13ff684a87f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:11.000512 12353 consensus_queue.cc:1035] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6f703b82d74f7386ca38a98137a397" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 40875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:11.000607 12330 consensus_queue.cc:1035] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d347df3a1f4e10800d419fe88fbcf7" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:11.010200 11810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:11.010314 11810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:11.010854 11810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03078dcf10f04af4b2e4ad48baf084e7: Sending DeleteTablet for 3 replicas of tablet 24d92cb09549478391f517c5069d3168
I20260925 07:21:11.011102 11949 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d92cb09549478391f517c5069d3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:11 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260925 07:21:11.011121 12229 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d92cb09549478391f517c5069d3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:11 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 07:21:11.011147 12089 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d92cb09549478391f517c5069d3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:11 UTC) from {username='oryx', principal='oryx/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:45750
I20260925 07:21:11.011197 12348 tablet_replica.cc:330] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: stopping tablet replica
I20260925 07:21:11.011214 12346 tablet_replica.cc:330] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: stopping tablet replica
I20260925 07:21:11.011219 12347 tablet_replica.cc:330] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: stopping tablet replica
I20260925 07:21:11.011257 12346 raft_consensus.cc:2229] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:11.011261 12347 raft_consensus.cc:2229] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:11.011262 12348 raft_consensus.cc:2229] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:11.011292 12347 raft_consensus.cc:2258] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:11.011298 12348 raft_consensus.cc:2258] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:11.011312 12346 raft_consensus.cc:2258] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:11.011624 12348 ts_tablet_manager.cc:1905] T 24d92cb09549478391f517c5069d3168 P d9d347df3a1f4e10800d419fe88fbcf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:11.011641 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11884
I20260925 07:21:11.012717 12346 ts_tablet_manager.cc:1905] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:11.014094 12346 ts_tablet_manager.cc:1918] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:21:11.014086 12347 ts_tablet_manager.cc:1905] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:11.014145 12346 log.cc:1196] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/24d92cb09549478391f517c5069d3168
I20260925 07:21:11.014353 12346 ts_tablet_manager.cc:1939] T 24d92cb09549478391f517c5069d3168 P 02bfad965606499ba6f13ff684a87f19: Deleting consensus metadata
I20260925 07:21:11.014672 11796 catalog_manager.cc:4841] TS 02bfad965606499ba6f13ff684a87f19 (127.4.47.3:37463): tablet 24d92cb09549478391f517c5069d3168 (table test-table [id=d258ffa3fc334dfe9afed2720587f3fe]) successfully deleted
I20260925 07:21:11.015515 12347 ts_tablet_manager.cc:1918] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:21:11.015591 12347 log.cc:1196] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/24d92cb09549478391f517c5069d3168
I20260925 07:21:11.015805 12347 ts_tablet_manager.cc:1939] T 24d92cb09549478391f517c5069d3168 P 2f6f703b82d74f7386ca38a98137a397: Deleting consensus metadata
I20260925 07:21:11.016151 11799 catalog_manager.cc:4841] TS 2f6f703b82d74f7386ca38a98137a397 (127.4.47.2:40875): tablet 24d92cb09549478391f517c5069d3168 (table test-table [id=d258ffa3fc334dfe9afed2720587f3fe]) successfully deleted
W20260925 07:21:11.017175 11799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:46583 (error 108)
I20260925 07:21:11.017477 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12024
I20260925 07:21:11.018146 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:11.017512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:46583 (local address 127.0.0.1:60238)
0925 07:21:11.018052 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:11.018098 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":56,"threads_started":1}
W20260925 07:21:11.018417 11799 catalog_manager.cc:4584] TS d9d347df3a1f4e10800d419fe88fbcf7 (127.4.47.1:46583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24d92cb09549478391f517c5069d3168: Network error: Client connection negotiation failed: client connection to 127.4.47.1:46583: connect: Connection refused (error 111)
I20260925 07:21:11.024228 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12164
I20260925 07:21:11.030045 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 11780
2026-09-25T07:21:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2533 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:11 dist-test-slave-zw07 krb5kdc[12408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:11 dist-test-slave-zw07 krb5kdc[12408](info): set up 2 sockets
Sep 25 07:21:11 dist-test-slave-zw07 krb5kdc[12408](info): commencing operation
krb5kdc: starting...
W20260925 07:21:12.067669 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:12 dist-test-slave-zw07 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:21:12.186753 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38157
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:12.265308 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:12.265517 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:12.265539 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:12.266717 12421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:12.266758 12421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:12.266773 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:12.266832 12421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:12.266857 12421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:12.268318 12421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:12.268579 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:12.268807 12421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:12.273728 12426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:12.273823 12421 server_base.cc:1034] running on GCE node
W20260925 07:21:12.273716 12427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:12.273731 12429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:12.274195 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:12.274492 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:12.275660 12421 hybrid_clock.cc:648] HybridClock initialized: now 1790320872275653 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T07:21:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:12 dist-test-slave-zw07 krb5kdc[12434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:12 dist-test-slave-zw07 krb5kdc[12434](info): set up 2 sockets
Sep 25 07:21:12 dist-test-slave-zw07 krb5kdc[12434](info): commencing operation
krb5kdc: starting...
W20260925 07:21:13.319758 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:21:13.441912 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40921
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:13.522826 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:13.523119 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:13.523168 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:13.524372 12447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:13.524448 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:13.524477 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:13.524509 12447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:13.524536 12447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:13.526021 12447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:40921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:13.526298 12447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:13.526530 12447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:13.531350 12452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:13.531359 12453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:13.531327 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:13.531642 12447 server_base.cc:1034] running on GCE node
I20260925 07:21:13.531893 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:13.532184 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:13.533354 12447 hybrid_clock.cc:648] HybridClock initialized: now 1790320873533341 us; error 38 us; skew 500 ppm
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:13.535909 12447 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:21:13.536367 12447 webserver.cc:460] Webserver started at http://127.4.47.62:38243/ using document root <none> and password file <none>
I20260925 07:21:13.536603 12447 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:13.536669 12447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:13.536815 12447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:13.537856 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "6bd9858c53364d049744bb7017c01c3b"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "97cb3cf4dafa9468a5d2c088f47c6953"
server_key_iv: "d33a27927108510c5ef4311b9aca66cb"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.538244 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "6bd9858c53364d049744bb7017c01c3b"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "97cb3cf4dafa9468a5d2c088f47c6953"
server_key_iv: "d33a27927108510c5ef4311b9aca66cb"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.539690 12447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:13.540398 12462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.540571 12447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:13.540678 12447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "6bd9858c53364d049744bb7017c01c3b"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "97cb3cf4dafa9468a5d2c088f47c6953"
server_key_iv: "d33a27927108510c5ef4311b9aca66cb"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.540752 12447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:13.549033 12447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:13.550835 12447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:13.551008 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:13.554740 12447 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:40921
I20260925 07:21:13.554771 12514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:40921 every 8 connection(s)
I20260925 07:21:13.555184 12447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:13.556061 12515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:13.556545 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12447
I20260925 07:21:13.556668 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:13.556840 4284 external_mini_cluster.cc:1428] Setting key bde116def0d0be428ff8eaa2de564379
I20260925 07:21:13.559258 12515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Bootstrap starting.
I20260925 07:21:13.559828 12515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:13.560119 12515 log.cc:824] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:13.560885 12515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: No bootstrap required, opened a new log
I20260925 07:21:13.563752 12515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:13.563905 12515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:13.563948 12515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd9858c53364d049744bb7017c01c3b, State: Initialized, Role: FOLLOWER
I20260925 07:21:13.564102 12515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:13.564175 12515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:13.564235 12515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:13.564285 12515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:13.565002 12515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:13.565109 12515 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd9858c53364d049744bb7017c01c3b; no voters:
I20260925 07:21:13.565303 12515 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:13.565338 12520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:13.565454 12520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 LEADER]: Becoming Leader. State: Replica: 6bd9858c53364d049744bb7017c01c3b, State: Running, Role: LEADER
I20260925 07:21:13.565533 12520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:13.565593 12515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:13.565850 12520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd9858c53364d049744bb7017c01c3b. Latest consensus state: current_term: 1 leader_uuid: "6bd9858c53364d049744bb7017c01c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } } }
I20260925 07:21:13.565923 12520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:13.565886 12522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd9858c53364d049744bb7017c01c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } } }
I20260925 07:21:13.565989 12522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:13.566123 12525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:13.566653 12525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:13.568588 12525 catalog_manager.cc:1347] Generated new cluster ID: d08fb7e1863147cda7c0465447509ea7
I20260925 07:21:13.568646 12525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:13.569247 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:13.557454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.4.47.62:40921)
0925 07:21:13.557616 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:13.557623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:13.557648 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:21:13.558058 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:13.558064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:13.558111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:13.558227 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:13.559056 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.560105 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.561107 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.561526 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.561610 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:13.564685 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:13.564689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:13.564690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:13.564703 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:13.567239 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.567692 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.567696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.567697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.567845 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.568084 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.568085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.568085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.568382 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:13.568524 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:13.568924 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 07:21:13.569060 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 07:21:13.592547 12525 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:13.593338 12525 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:13.596249 12525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Generated new TSK 0
I20260925 07:21:13.596453 12525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:21:13.630649 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:13.712317 12541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:13.712534 12541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:13.712558 12541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:13.713671 12541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:13.713706 12541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:13.713730 12541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:13.715210 12541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:13.715399 12541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:13.715583 12541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:13.720467 12549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:13.720481 12546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:13.720655 12541 server_base.cc:1034] running on GCE node
W20260925 07:21:13.720481 12547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:13.720937 12541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:13.721218 12541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:13.722406 12541 hybrid_clock.cc:648] HybridClock initialized: now 1790320873722388 us; error 55 us; skew 500 ppm
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:13.724874 12541 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:21:13.725363 12541 webserver.cc:460] Webserver started at http://127.4.47.1:39431/ using document root <none> and password file <none>
I20260925 07:21:13.725618 12541 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:13.725687 12541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:13.725840 12541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:13.726912 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "a623e9ee0fcb461882c78abef85eeed0"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "100beef47d4cc372cec921c9db138cfb"
server_key_iv: "b03fa4907ddbe7611ccb420266f7ad3d"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.727305 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "a623e9ee0fcb461882c78abef85eeed0"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "100beef47d4cc372cec921c9db138cfb"
server_key_iv: "b03fa4907ddbe7611ccb420266f7ad3d"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.728564 12541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:13.729264 12556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.729429 12541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:13.729503 12541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "a623e9ee0fcb461882c78abef85eeed0"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "100beef47d4cc372cec921c9db138cfb"
server_key_iv: "b03fa4907ddbe7611ccb420266f7ad3d"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.729557 12541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:13.737066 12541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:13.738844 12541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:13.738965 12541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:13.739238 12541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:13.739512 12541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:13.739544 12541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.739565 12541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:13.739578 12541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.745391 12541 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:41665
I20260925 07:21:13.745479 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:41665 every 8 connection(s)
I20260925 07:21:13.745815 12541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:13.755388 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:13.746552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:32863 (local address 127.4.47.62:40921)
0925 07:21:13.746626 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:13.746629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:13.746898 (+ 269us) server_negotiation.cc:407] Connection header received
0925 07:21:13.747425 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:13.747425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:13.747436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:13.747510 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:13.748481 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.748978 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.749923 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.750119 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.750173 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:13.752370 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:13.752371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:13.752372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:13.752377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:13.754075 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.754532 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.754533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.754534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.754586 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.754801 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.754802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.754802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.754904 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:13.754956 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:13.755244 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 07:21:13.755323 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:21:13.755589 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:13.746541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:32863)
0925 07:21:13.746734 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:13.746789 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:13.747334 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:13.747580 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:13.747583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:13.748194 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:13.748386 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:13.748390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.749056 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.749056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:13.749849 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:13.749851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.750192 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.750193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:13.750288 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:13.750844 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:13.750859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:13.752287 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:13.754178 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:13.754185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:13.754188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:13.754462 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:13.754650 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:13.754651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:13.754651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:13.754748 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:13.754962 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:13.754966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:13.755132 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 07:21:13.755217 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 07:21:13.755342 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
I20260925 07:21:13.755677 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12541
I20260925 07:21:13.755793 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:13.756031 4284 external_mini_cluster.cc:1428] Setting key 3a21c4de5766e958e4e30be3f139a6d1
I20260925 07:21:13.756233 12670 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40921
I20260925 07:21:13.756345 12670 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:13.756541 12670 heartbeater.cc:511] Master 127.4.47.62:40921 requested a full tablet report, sending...
I20260925 07:21:13.757072 12479 ts_manager.cc:194] Registered new tserver with Master: a623e9ee0fcb461882c78abef85eeed0 (127.4.47.1:41665)
I20260925 07:21:13.757682 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:32863
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:21:13.781840 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:13.861933 12675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:13.862141 12675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:13.862164 12675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:13.863366 12675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:13.863405 12675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:13.863464 12675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:13.865407 12675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:13.865689 12675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:13.865974 12675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:13.871502 12683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:13.871513 12681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:13.871519 12680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:13.871695 12675 server_base.cc:1034] running on GCE node
I20260925 07:21:13.871898 12675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:13.872150 12675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:13.873287 12675 hybrid_clock.cc:648] HybridClock initialized: now 1790320873873280 us; error 30 us; skew 500 ppm
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:13.875824 12675 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:21:13.876291 12675 webserver.cc:460] Webserver started at http://127.4.47.2:44361/ using document root <none> and password file <none>
I20260925 07:21:13.876520 12675 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:13.876564 12675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:13.876701 12675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:13.877753 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "c13d295c59df43c1a88c8fb25ba1cf5e"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "aaf37d3e5ca5ec6500d06341fe0d7e39"
server_key_iv: "6b8ea25eb0f8fd2e0c02178361b8327a"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.878108 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "c13d295c59df43c1a88c8fb25ba1cf5e"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "aaf37d3e5ca5ec6500d06341fe0d7e39"
server_key_iv: "6b8ea25eb0f8fd2e0c02178361b8327a"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.879472 12675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:13.880223 12690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.880450 12675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:13.880539 12675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "c13d295c59df43c1a88c8fb25ba1cf5e"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "aaf37d3e5ca5ec6500d06341fe0d7e39"
server_key_iv: "6b8ea25eb0f8fd2e0c02178361b8327a"
server_key_version: "encryptionkey@0"
I20260925 07:21:13.880609 12675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:13.893985 12675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:13.896099 12675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:13.896255 12675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:13.896548 12675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:13.896883 12675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:13.896917 12675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.896971 12675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:13.896997 12675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:13.903093 12675 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:36381
I20260925 07:21:13.903182 12803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:36381 every 8 connection(s)
I20260925 07:21:13.903522 12675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:13.906711 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12675
I20260925 07:21:13.906806 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:13.907008 4284 external_mini_cluster.cc:1428] Setting key 80d95714768fc64f2afa496bd4275413
Sep 25 07:21:13 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320873, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:13.913811 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:13.904327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55085 (local address 127.4.47.62:40921)
0925 07:21:13.904376 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:13.904379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:13.904673 (+ 294us) server_negotiation.cc:407] Connection header received
0925 07:21:13.905165 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:13.905167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:13.905177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:13.905232 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:13.906266 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.906840 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.907976 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.908185 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.908264 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:13.910837 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:13.910838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:13.910839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:13.910847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:13.912305 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.912773 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.912775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.912775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.912835 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:13.913099 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:13.913100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:13.913100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:13.913217 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:13.913305 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:13.913636 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 07:21:13.913714 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:21:13.914029 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:13.904314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:55085)
0925 07:21:13.904531 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:13.904581 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:13.905075 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:13.905286 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:13.905290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:13.905880 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:13.906177 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:13.906183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.906941 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.906942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:13.907868 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:13.907870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:13.908272 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:13.908273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:13.908408 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:13.909006 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:13.909016 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:13.910736 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:13.912432 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:13.912439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:13.912441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:13.912680 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:13.912904 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:13.912905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:13.912905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:13.913024 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:13.913297 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:13.913303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:13.913493 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 07:21:13.913605 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 07:21:13.913778 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260925 07:21:13.914518 12804 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40921
I20260925 07:21:13.914616 12804 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
I20260925 07:21:13.914844 12804 heartbeater.cc:511] Master 127.4.47.62:40921 requested a full tablet report, sending...
I20260925 07:21:13.915290 12479 ts_manager.cc:194] Registered new tserver with Master: c13d295c59df43c1a88c8fb25ba1cf5e (127.4.47.2:36381)
I20260925 07:21:13.915637 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:55085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:21:13.934065 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:14.014976 12809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:14.015218 12809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:14.015244 12809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:14.016388 12809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:14.016426 12809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:14.016453 12809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:14.017822 12809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:14.018020 12809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:14.018258 12809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:14.023380 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.023375 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.023375 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.023591 12809 server_base.cc:1034] running on GCE node
I20260925 07:21:14.023844 12809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:14.024099 12809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:14.025265 12809 hybrid_clock.cc:648] HybridClock initialized: now 1790320874025253 us; error 32 us; skew 500 ppm
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:14.027936 12809 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:21:14.028537 12809 webserver.cc:460] Webserver started at http://127.4.47.3:34647/ using document root <none> and password file <none>
I20260925 07:21:14.028777 12809 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:14.028844 12809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:14.028990 12809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:14.030076 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "463620ee2eac473891837eec372ab725"
format_stamp: "Formatted at 2026-09-25 07:21:14 on dist-test-slave-zw07"
server_key: "458f907fa476dd28f9a6e6f02dedeebe"
server_key_iv: "5b554ea67f4bc03347814e081ee62087"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.030443 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "463620ee2eac473891837eec372ab725"
format_stamp: "Formatted at 2026-09-25 07:21:14 on dist-test-slave-zw07"
server_key: "458f907fa476dd28f9a6e6f02dedeebe"
server_key_iv: "5b554ea67f4bc03347814e081ee62087"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.031962 12809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:14.033005 12824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.033185 12809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:14.033298 12809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "463620ee2eac473891837eec372ab725"
format_stamp: "Formatted at 2026-09-25 07:21:14 on dist-test-slave-zw07"
server_key: "458f907fa476dd28f9a6e6f02dedeebe"
server_key_iv: "5b554ea67f4bc03347814e081ee62087"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.033376 12809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:14.047281 12809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:14.049470 12809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:14.049641 12809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:14.049878 12809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:14.050189 12809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:14.050243 12809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.050285 12809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:14.050313 12809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.056259 12809 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:38047
I20260925 07:21:14.056303 12937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:38047 every 8 connection(s)
I20260925 07:21:14.056649 12809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:14.058900 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12809
I20260925 07:21:14.059036 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:14.059243 4284 external_mini_cluster.cc:1428] Setting key 6fa5ba558e5cf702d38cccda07c7c494
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:14.066147 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.057421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:39291 (local address 127.4.47.62:40921)
0925 07:21:14.057498 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.057501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.057687 (+ 186us) server_negotiation.cc:407] Connection header received
0925 07:21:14.058207 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.058208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.058220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.058271 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.059351 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.060136 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.061173 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.061369 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.061427 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.063614 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:14.063616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:14.063616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:14.063622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:14.064827 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:14.065186 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:14.065187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:14.065188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:14.065239 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:14.065489 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:14.065489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:14.065490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:14.065623 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:14.065675 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:14.065925 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 07:21:14.065994 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:21:14.066187 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.057410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:39291)
0925 07:21:14.057568 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.057605 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.058104 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.058357 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.058361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.058926 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.059233 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.059239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.060230 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.060231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.061091 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.061094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.061456 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.061457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.061554 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.062114 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.062123 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.063535 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:14.064897 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:14.064903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:14.064905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:14.065122 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:14.065300 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:14.065301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:14.065301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:14.065404 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:14.065683 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:14.065688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:14.065810 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 07:21:14.065897 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:21:14.066022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260925 07:21:14.066792 12938 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40921
I20260925 07:21:14.066896 12938 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:14.067139 12938 heartbeater.cc:511] Master 127.4.47.62:40921 requested a full tablet report, sending...
I20260925 07:21:14.067545 12479 ts_manager.cc:194] Registered new tserver with Master: 463620ee2eac473891837eec372ab725 (127.4.47.3:38047)
I20260925 07:21:14.067948 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:39291
I20260925 07:21:14.070931 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
Entry for principal oryx/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab.
WARNING: no policy specified for oryx/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
Entry for principal oryx/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab.
WARNING: no policy specified for oryx/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
Entry for principal oryx/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab.
I20260925 07:21:14.141848 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12541
I20260925 07:21:14.146756 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12675
I20260925 07:21:14.151876 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12809
I20260925 07:21:14.156594 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12447
I20260925 07:21:14.162549 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40921
--webserver_interface=127.4.47.62
--webserver_port=38243
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:14.243718 12947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:14.243980 12947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:14.244022 12947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:14.245203 12947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:14.245287 12947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:14.245316 12947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:14.245347 12947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:14.245370 12947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:14.246879 12947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:40921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:14.247196 12947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:14.247433 12947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:14.252280 12952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.252269 12955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.252269 12953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.252640 12947 server_base.cc:1034] running on GCE node
I20260925 07:21:14.252897 12947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:14.253188 12947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:14.254336 12947 hybrid_clock.cc:648] HybridClock initialized: now 1790320874254308 us; error 45 us; skew 500 ppm
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:14.256856 12947 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:21:14.257423 12947 webserver.cc:460] Webserver started at http://127.4.47.62:38243/ using document root <none> and password file <none>
I20260925 07:21:14.257678 12947 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:14.257759 12947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:14.259240 12947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:14.259963 12962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.260159 12947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:14.260262 12947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "6bd9858c53364d049744bb7017c01c3b"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "97cb3cf4dafa9468a5d2c088f47c6953"
server_key_iv: "d33a27927108510c5ef4311b9aca66cb"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.260633 12947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:14.269899 12947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:14.271975 12947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:14.272131 12947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:14.276603 12947 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:40921
I20260925 07:21:14.276633 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:40921 every 8 connection(s)
I20260925 07:21:14.277012 12947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:14.278419 13015 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:21:14.279121 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Bootstrap starting.
I20260925 07:21:14.286979 13015 log.cc:824] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:14.287315 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 12947
I20260925 07:21:14.287639 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:41665
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:40921
--webserver_port=39431
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:21:14.289348 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:21:14.289666 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Bootstrap complete.
I20260925 07:21:14.293483 13015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:14.293857 13015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd9858c53364d049744bb7017c01c3b, State: Initialized, Role: FOLLOWER
I20260925 07:21:14.294088 13015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:14.294178 13015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:14.294229 13015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:14.294273 13015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:21:14.295812 13015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:14.295951 13015 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd9858c53364d049744bb7017c01c3b; no voters:
I20260925 07:21:14.296144 13015 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:21:14.296252 13019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:21:14.296452 13019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [term 2 LEADER]: Becoming Leader. State: Replica: 6bd9858c53364d049744bb7017c01c3b, State: Running, Role: LEADER
I20260925 07:21:14.296506 13015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:14.296623 13019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } }
I20260925 07:21:14.296932 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd9858c53364d049744bb7017c01c3b. Latest consensus state: current_term: 2 leader_uuid: "6bd9858c53364d049744bb7017c01c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } } }
I20260925 07:21:14.296936 13020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6bd9858c53364d049744bb7017c01c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd9858c53364d049744bb7017c01c3b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40921 } } }
I20260925 07:21:14.297199 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:14.297219 13020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:14.297709 13029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:14.298341 13029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:14.298506 13029 catalog_manager.cc:1259] Loaded cluster ID: d08fb7e1863147cda7c0465447509ea7
I20260925 07:21:14.298557 13029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:14.299983 13029 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:14.300184 13029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6bd9858c53364d049744bb7017c01c3b: Loaded TSK: 0
I20260925 07:21:14.300405 13029 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:21:14.373719 13018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:14.373926 13018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:14.373958 13018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:14.375239 13018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:14.375283 13018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:14.375310 13018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:14.376750 13018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.1.keytab
--principal=oryx/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:41665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=39431
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:14.377000 13018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:14.377250 13018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:14.382257 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.382313 13042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.382247 13041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.382560 13018 server_base.cc:1034] running on GCE node
I20260925 07:21:14.382753 13018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:14.383015 13018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:14.384222 13018 hybrid_clock.cc:648] HybridClock initialized: now 1790320874384202 us; error 27 us; skew 500 ppm
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:14.387290 13018 init.cc:377] Logged in from keytab as oryx/127.4.47.1@KRBTEST.COM (short username oryx)
I20260925 07:21:14.387758 13018 webserver.cc:460] Webserver started at http://127.4.47.1:39431/ using document root <none> and password file <none>
I20260925 07:21:14.388010 13018 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:14.388093 13018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:14.389506 13018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 07:21:14.390147 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.390316 13018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:21:14.390419 13018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "a623e9ee0fcb461882c78abef85eeed0"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "100beef47d4cc372cec921c9db138cfb"
server_key_iv: "b03fa4907ddbe7611ccb420266f7ad3d"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.390756 13018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:14.411288 13018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:14.413501 13018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:14.413648 13018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:14.413925 13018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:14.414273 13018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:14.414311 13018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.414335 13018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:14.414378 13018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.421595 13018 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:41665
I20260925 07:21:14.421634 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:41665 every 8 connection(s)
I20260925 07:21:14.421980 13018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:21:14.424065 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13018
I20260925 07:21:14.424377 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:36381
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:40921
--webserver_port=44361
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.430868 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.422838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:37617)
0925 07:21:14.423022 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.423099 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.423622 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.424072 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.424076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.424792 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.424984 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.424988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.426074 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.426075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.426877 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.426879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.427569 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.427570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.427704 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.428271 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.428279 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.430640 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
W20260925 07:21:14.430907 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.422889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:37617 (local address 127.4.47.62:40921)
0925 07:21:14.423158 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.423165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.423201 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:21:14.423831 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.423839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.423888 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.424035 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.425104 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.425970 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.426975 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.427468 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.427602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.430763 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:37617: BlockingRecv error: recv got EOF from 127.4.47.1:37617 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
W20260925 07:21:14.431481 13165 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:40921 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.47.62:40921: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.435873 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.431727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:48605)
0925 07:21:14.431762 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.431780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.431831 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.431947 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.431948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.432080 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.432123 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.432124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.432763 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.432764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.433257 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.433259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.433629 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.433630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.433702 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.434153 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.434157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.435792 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 07:21:14.435925 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.431787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:48605 (local address 127.4.47.62:40921)
0925 07:21:14.431849 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.431851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.431859 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:14.431884 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.431885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.431897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.431967 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.432207 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.432680 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.433337 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.433545 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.433629 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.435864 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:48605: BlockingRecv error: recv got EOF from 127.4.47.1:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.440022 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.436115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:51383)
0925 07:21:14.436154 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.436165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.436200 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.436340 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.436340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.436465 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.436489 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.436490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.437100 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.437101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.437586 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.437587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.437930 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.437931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.437988 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.438387 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.438391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.439956 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 07:21:14.440152 13165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:21:14.440235 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.436169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:51383 (local address 127.4.47.62:40921)
0925 07:21:14.436201 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.436203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.436211 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:14.436253 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.436254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.436264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.436341 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.436568 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.437008 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.437665 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.437847 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.437928 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.440033 (+ 2105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:51383: BlockingRecv error: recv got EOF from 127.4.47.1:51383 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 07:21:14.506620 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:14.506829 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:14.506877 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:14.508107 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:14.508152 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:14.508178 13169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:14.509626 13169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.2.keytab
--principal=oryx/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:36381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=44361
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:14.509812 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:14.509984 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:14.515214 13175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.515230 13177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.515231 13174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.515259 13169 server_base.cc:1034] running on GCE node
I20260925 07:21:14.515693 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:14.515930 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:14.517091 13169 hybrid_clock.cc:648] HybridClock initialized: now 1790320874517071 us; error 33 us; skew 500 ppm
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:14.519686 13169 init.cc:377] Logged in from keytab as oryx/127.4.47.2@KRBTEST.COM (short username oryx)
I20260925 07:21:14.520151 13169 webserver.cc:460] Webserver started at http://127.4.47.2:44361/ using document root <none> and password file <none>
I20260925 07:21:14.520391 13169 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:14.520504 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:14.521921 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 07:21:14.522506 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.522676 13169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.522773 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "c13d295c59df43c1a88c8fb25ba1cf5e"
format_stamp: "Formatted at 2026-09-25 07:21:13 on dist-test-slave-zw07"
server_key: "aaf37d3e5ca5ec6500d06341fe0d7e39"
server_key_iv: "6b8ea25eb0f8fd2e0c02178361b8327a"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.523201 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:14.533658 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:14.535609 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:14.535718 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:14.535897 13169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:14.536139 13169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:14.536167 13169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.536190 13169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:14.536201 13169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.541975 13169 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:36381
I20260925 07:21:14.542050 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:36381 every 8 connection(s)
I20260925 07:21:14.542366 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.549201 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13169
I20260925 07:21:14.549481 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:38047
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:40921
--webserver_port=34647
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:14.549520 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.543237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:52915 (local address 127.4.47.62:40921)
0925 07:21:14.543308 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.543311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.543548 (+ 237us) server_negotiation.cc:407] Connection header received
0925 07:21:14.544014 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.544015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.544028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.544080 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.544969 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.545451 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.546381 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.546586 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.546639 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.549466 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:52915: BlockingRecv error: recv got EOF from 127.4.47.2:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:21:14.549578 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.543240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:52915)
0925 07:21:14.543422 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.543460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.543924 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.544135 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.544138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.544695 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.544890 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.544895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.545529 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.545530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.546310 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.546312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.546658 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.546658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.546752 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.547302 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.547310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.549384 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260925 07:21:14.549811 13298 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:40921 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.47.62:40921: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.554203 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.550051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:53549)
0925 07:21:14.550124 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.550135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.550181 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.550326 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.550326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.550462 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.550502 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.550503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.551157 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.551158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.551661 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.551662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.552038 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.552039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.552107 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.552548 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.552553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.554156 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 07:21:14.554360 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.550088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:53549 (local address 127.4.47.62:40921)
0925 07:21:14.550128 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.550130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.550167 (+ 37us) server_negotiation.cc:407] Connection header received
0925 07:21:14.550241 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.550242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.550251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.550324 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.550589 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.551042 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.551763 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.551953 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.552037 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.554279 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:53549: BlockingRecv error: recv got EOF from 127.4.47.2:53549 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.558956 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.554507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:50769)
0925 07:21:14.554547 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.554556 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.554588 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.554717 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.554718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.554815 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.554840 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.554841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.555784 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.555785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.556246 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.556247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.556733 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.556734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.556767 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.557179 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.557183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.558895 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 07:21:14.559103 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.554556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:50769 (local address 127.4.47.62:40921)
0925 07:21:14.554590 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.554592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.554601 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:14.554632 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.554633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.554644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.554716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.554909 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.555708 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.556324 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.556655 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.556733 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.559003 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:50769: BlockingRecv error: recv got EOF from 127.4.47.2:50769 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:21:14.559180 13298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:21:14.633165 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:14.633385 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:14.633409 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:14.634502 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:14.634536 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:14.634562 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:14.635938 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/krb5kdc/oryx_127.4.47.3.keytab
--principal=oryx/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:38047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=34647
--tserver_master_addrs=127.4.47.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:14.636129 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:14.636317 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:14.641393 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:14.641372 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.641507 13301 server_base.cc:1034] running on GCE node
W20260925 07:21:14.641376 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:14.641816 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:14.642040 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:14.643199 13301 hybrid_clock.cc:648] HybridClock initialized: now 1790320874643183 us; error 26 us; skew 500 ppm
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320874, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:14.645802 13301 init.cc:377] Logged in from keytab as oryx/127.4.47.3@KRBTEST.COM (short username oryx)
I20260925 07:21:14.646257 13301 webserver.cc:460] Webserver started at http://127.4.47.3:34647/ using document root <none> and password file <none>
I20260925 07:21:14.646458 13301 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:14.646508 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:14.647900 13301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:14.648458 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.648648 13301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:14.648730 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "463620ee2eac473891837eec372ab725"
format_stamp: "Formatted at 2026-09-25 07:21:14 on dist-test-slave-zw07"
server_key: "458f907fa476dd28f9a6e6f02dedeebe"
server_key_iv: "5b554ea67f4bc03347814e081ee62087"
server_key_version: "encryptionkey@0"
I20260925 07:21:14.649065 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:14.668306 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:14.670193 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:14.670308 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:14.670521 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:14.670852 13301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:14.670888 13301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.670912 13301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:14.670960 13301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:14.676788 13301 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:38047
I20260925 07:21:14.676832 13429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:38047 every 8 connection(s)
I20260925 07:21:14.677199 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:21:14.684453 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.678000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:53573 (local address 127.4.47.62:40921)
0925 07:21:14.678075 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.678078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.678293 (+ 215us) server_negotiation.cc:407] Connection header received
0925 07:21:14.678761 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.678762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.678771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.678823 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.679846 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.680328 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.681285 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.681478 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.681531 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.684399 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:53573: BlockingRecv error: recv got EOF from 127.4.47.3:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:21:14.684508 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.677984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:53573)
0925 07:21:14.678170 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.678207 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.678676 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.678881 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.678884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.679551 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.679752 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.679758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.680409 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.680410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.681208 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.681210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.681552 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.681553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.681650 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.682133 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.682141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.684314 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260925 07:21:14.684690 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13301
W20260925 07:21:14.684752 13430 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:40921 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.47.62:40921: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.689544 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.684993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:37473)
0925 07:21:14.685043 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.685054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.685094 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.685354 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.685355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.685482 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.685524 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.685525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.686174 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.686175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.686670 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.686672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.687046 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.687047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.687133 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.687551 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.687556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.689461 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:21:14.689647 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.685197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:37473 (local address 127.4.47.62:40921)
0925 07:21:14.685252 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.685255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.685262 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:14.685286 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.685287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.685297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.685349 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.685608 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.686085 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.686765 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.686964 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.687046 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.689579 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:37473: BlockingRecv error: recv got EOF from 127.4.47.3:37473 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:21:14.692482 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.685216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.4.47.62:40921)
0925 07:21:14.685342 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.685344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.685353 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:14.685425 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.685426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.685437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.685521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.686075 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.686541 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.687404 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.687741 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.687825 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.689055 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:14.689065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:14.689066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:14.689077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:14.691036 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:14.691490 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:14.691493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:14.691493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:14.691584 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:14.691779 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:14.691779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:14.691779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:14.692032 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:14.692157 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:14.692246 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 07:21:14.692365 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:14 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:14.694527 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:14.689913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:53047)
0925 07:21:14.689982 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:14.689993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:14.690036 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:14.690170 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:14.690171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:14.690306 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:14.690333 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.690334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.691209 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.691210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.691920 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:14.691922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.692333 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.692334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:14.692388 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.692837 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:14.692841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:14.694484 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 07:21:14.694623 13168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:14.689995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:53047 (local address 127.4.47.62:40921)
0925 07:21:14.690044 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:14.690046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:14.690056 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:14.690081 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:14.690082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:14.690091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:14.690145 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:14.690408 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.691139 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.691995 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:14.692257 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:14.692318 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:14.694559 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:53047: BlockingRecv error: recv got EOF from 127.4.47.3:53047 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 07:21:14.694703 13430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:15.449680 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:15.440587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:33737)
0925 07:21:15.440730 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:15.440740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:15.440786 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:15.440912 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:15.440913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:15.441067 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:15.441115 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.441117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.441750 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.441750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.442303 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.442304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.442632 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.442633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.442672 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.447839 (+ 5167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:15.447844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:15.449612 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260925 07:21:15.449822 13436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:15.440673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:33737 (local address 127.4.47.62:40921)
0925 07:21:15.440794 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:15.440795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:15.440804 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:15.440826 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:15.440827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:15.440835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:15.440895 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:15.441179 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.441684 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.442375 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.442570 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.442620 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.449747 (+ 7127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:33737: BlockingRecv error: recv got EOF from 127.4.47.1:33737 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:15.564209 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:15.559563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:51009)
0925 07:21:15.559706 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:15.559724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:15.559789 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:15.559935 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:15.559935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:15.560098 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:15.560148 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.560149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.560790 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.560791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.561343 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.561345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.561688 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.561689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.561741 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.562189 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:15.562194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:15.564141 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
W20260925 07:21:15.564312 13436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:15.559671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:51009 (local address 127.4.47.62:40921)
0925 07:21:15.559741 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:15.559743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:15.559777 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:21:15.559848 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:15.559848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:15.559858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:15.559914 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:15.560213 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.560721 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.561427 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.561622 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.561673 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.564241 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:51009: BlockingRecv error: recv got EOF from 127.4.47.2:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:15 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:15.700249 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:15.695170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:54435)
0925 07:21:15.695308 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:15.695327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:15.695389 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:15.695562 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:15.695564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:15.695734 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:15.695784 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.695786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.696501 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.696502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.697274 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:15.697277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.697783 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.697785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:15.697851 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.698373 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:15.698377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:15.700172 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260925 07:21:15.700353 13436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:15.695203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:54435 (local address 127.4.47.62:40921)
0925 07:21:15.695272 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:15.695274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:15.695388 (+ 114us) server_negotiation.cc:407] Connection header received
0925 07:21:15.695433 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:15.695434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:15.695446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:15.695507 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:15.695854 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.696403 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.697390 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:15.697683 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:15.697782 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:15.700267 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:54435: BlockingRecv error: recv got EOF from 127.4.47.3:54435 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:16.454876 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:16.450132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:52385)
0925 07:21:16.450286 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:16.450297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:16.450344 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:16.450536 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:16.450537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:16.450696 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:16.450744 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.450746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.451552 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.451552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.452140 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.452142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.452503 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.452504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.452547 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.452975 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:16.452979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:16.454819 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260925 07:21:16.455025 13440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:16.450254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52385 (local address 127.4.47.62:40921)
0925 07:21:16.450398 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:16.450399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:16.450407 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:16.450436 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:16.450437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:16.450447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:16.450520 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:16.450837 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.451475 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.452232 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.452432 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.452493 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.454921 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:52385: BlockingRecv error: recv got EOF from 127.4.47.1:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:16.569213 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:16.564695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:45765)
0925 07:21:16.564811 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:16.564823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:16.564896 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:16.565037 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:16.565038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:16.565199 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:16.565248 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.565250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.565871 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.565872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.566392 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.566393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.566771 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.566771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.566819 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.567405 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:16.567410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:16.569155 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
W20260925 07:21:16.569375 13440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:16.564771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:45765 (local address 127.4.47.62:40921)
0925 07:21:16.564836 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:16.564838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:16.564893 (+ 55us) server_negotiation.cc:407] Connection header received
0925 07:21:16.564942 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:16.564943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:16.564954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:16.565017 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:16.565308 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.565804 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.566484 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.566681 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.566749 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.569286 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:45765: BlockingRecv error: recv got EOF from 127.4.47.2:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:16 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:16.706912 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:16.700716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:35375)
0925 07:21:16.700881 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:16.700897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:16.700968 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:16.701192 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:16.701194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:16.701423 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:16.701483 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.701485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.702516 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.702518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.703363 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:16.703366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.703881 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.703882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:16.703938 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.704548 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:16.704554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:16.706821 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
W20260925 07:21:16.707185 13440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:16.700861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:35375 (local address 127.4.47.62:40921)
0925 07:21:16.700939 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:16.700941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:16.700950 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:16.701063 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:16.701064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:16.701081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:16.701139 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:16.701562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.702430 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.703470 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:16.703755 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:16.703822 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:16.707122 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:35375: BlockingRecv error: recv got EOF from 127.4.47.3:35375 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:17.459836 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:17.455385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:42565)
0925 07:21:17.455531 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:17.455548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:17.455599 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:17.455738 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:17.455738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:17.455893 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:17.455942 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.455943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.456592 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.456592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.457134 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.457136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.457459 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.457459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.457497 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.457996 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:17.458000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:17.459770 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260925 07:21:17.459915 13444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:17.455440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:42565 (local address 127.4.47.62:40921)
0925 07:21:17.455603 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:17.455605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:17.455638 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:21:17.455660 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:17.455660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:17.455671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:17.455722 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:17.456027 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.456525 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.457207 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.457393 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.457452 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.459861 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:42565: BlockingRecv error: recv got EOF from 127.4.47.1:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:17.574673 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:17.569880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:40983)
0925 07:21:17.570090 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:17.570102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:17.570188 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:17.570334 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:17.570335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:17.570531 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:17.570578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.570580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.571323 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.571324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.571874 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.571876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.572254 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.572255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.572310 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.572825 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:17.572835 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:17.574564 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":133,"threads_started":1}
W20260925 07:21:17.574726 13444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:17.569960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:40983 (local address 127.4.47.62:40921)
0925 07:21:17.570010 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:17.570013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:17.570173 (+ 160us) server_negotiation.cc:407] Connection header received
0925 07:21:17.570259 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:17.570260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:17.570275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:17.570327 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:17.570663 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.571239 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.571980 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.572183 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.572257 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.574656 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:40983: BlockingRecv error: recv got EOF from 127.4.47.2:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:17 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:17.712168 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:17.707445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:48511)
0925 07:21:17.707595 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:17.707610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:17.707677 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:17.707821 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:17.707822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:17.707998 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:17.708052 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.708054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.708697 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.708698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.709250 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:17.709251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.709573 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.709574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:17.709612 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.710076 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:17.710081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:17.712069 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
W20260925 07:21:17.712250 13444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:17.707558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:48511 (local address 127.4.47.62:40921)
0925 07:21:17.707626 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:17.707628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:17.707665 (+ 37us) server_negotiation.cc:407] Connection header received
0925 07:21:17.707734 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:17.707735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:17.707747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:17.707802 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:17.708124 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.708621 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.709327 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:17.709511 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:17.709564 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:17.712165 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:48511: BlockingRecv error: recv got EOF from 127.4.47.3:48511 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:18.465368 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:18.460456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:48465)
0925 07:21:18.460589 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:18.460605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:18.460671 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:18.460931 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:18.460933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:18.461143 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:18.461203 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.461205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.461873 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.461875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.462425 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.462426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.462819 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.462821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.462884 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.463515 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:18.463519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:18.465299 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260925 07:21:18.465528 13448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:18.460559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:48465 (local address 127.4.47.62:40921)
0925 07:21:18.460758 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:18.460759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:18.460766 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:21:18.460804 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:18.460804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:18.460817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:18.460885 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:18.461269 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.461796 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.462507 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.462690 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.462796 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.465431 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:48465: BlockingRecv error: recv got EOF from 127.4.47.1:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:18.580050 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:18.575229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:56419)
0925 07:21:18.575384 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:18.575402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:18.575466 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:18.575609 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:18.575609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:18.575804 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:18.575858 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.575860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.576549 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.576551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.577122 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.577124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.577558 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.577558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.577598 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.578062 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:18.578066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:18.579991 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260925 07:21:18.580214 13448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:18.575318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56419 (local address 127.4.47.62:40921)
0925 07:21:18.575387 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:18.575389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:18.575451 (+ 62us) server_negotiation.cc:407] Connection header received
0925 07:21:18.575514 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:18.575514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:18.575526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:18.575579 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:18.575946 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.576457 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.577224 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.577483 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.577550 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.580107 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:56419: BlockingRecv error: recv got EOF from 127.4.47.2:56419 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:18 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:18.717562 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:18.712631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:46171)
0925 07:21:18.712761 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:18.712772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:18.712839 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:18.712971 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:18.712972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:18.713130 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:18.713179 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.713181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.713797 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.713798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.714324 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:18.714326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.714676 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.714677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:18.714722 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.715274 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:18.715278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:18.717505 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
W20260925 07:21:18.717700 13448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:18.712717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:46171 (local address 127.4.47.62:40921)
0925 07:21:18.712764 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:18.712766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:18.712824 (+ 58us) server_negotiation.cc:407] Connection header received
0925 07:21:18.712885 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:18.712886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:18.712899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:18.712955 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:18.713241 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.713727 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.714419 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:18.714604 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:18.714661 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:18.717617 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:46171: BlockingRecv error: recv got EOF from 127.4.47.3:46171 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:19.470557 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:19.465823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:46085)
0925 07:21:19.465977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:19.465988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:19.466054 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:19.466201 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:19.466202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:19.466363 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:19.466411 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.466413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.467138 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.467138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.467687 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.467689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.468016 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.468017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.468055 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.468558 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:19.468563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:19.470500 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
W20260925 07:21:19.470670 13452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:19.465899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46085 (local address 127.4.47.62:40921)
0925 07:21:19.465998 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:19.466000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:19.466043 (+ 43us) server_negotiation.cc:407] Connection header received
0925 07:21:19.466116 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:19.466116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:19.466127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:19.466187 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:19.466513 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.467017 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.467762 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.467953 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.468005 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.470592 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:46085: BlockingRecv error: recv got EOF from 127.4.47.1:46085 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:19.585706 13453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:19.580702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:56081)
0925 07:21:19.580906 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:19.580927 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:19.581003 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:19.581154 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:19.581155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:19.581396 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:19.581448 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.581450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.582122 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.582123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.582667 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.582669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.583011 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.583012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.583053 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.583618 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:19.583623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:19.585647 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
W20260925 07:21:19.585817 13452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:19.580902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56081 (local address 127.4.47.62:40921)
0925 07:21:19.580984 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:19.580986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:19.580995 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:21:19.581065 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:19.581065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:19.581076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:19.581125 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:19.581521 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.582026 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.582744 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.582946 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.582996 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.585749 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:56081: BlockingRecv error: recv got EOF from 127.4.47.2:56081 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:19 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:19.723459 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:19.718141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:34215)
0925 07:21:19.718321 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:19.718336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:19.718428 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:19.718680 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:19.718682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:19.718926 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:19.718987 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.718990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.719884 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.719885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.720528 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:19.720531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.720922 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.720923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:19.720975 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.721496 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:19.721500 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:19.723402 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
W20260925 07:21:19.723619 13452 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:19.718296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:34215 (local address 127.4.47.62:40921)
0925 07:21:19.718357 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:19.718360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:19.718405 (+ 45us) server_negotiation.cc:407] Connection header received
0925 07:21:19.718526 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:19.718527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:19.718543 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:19.718621 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:19.719108 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.719750 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.720622 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:19.720832 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:19.720897 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:19.723534 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:34215: BlockingRecv error: recv got EOF from 127.4.47.3:34215 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:20.486112 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:20.471140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:37725)
0925 07:21:20.471289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:20.471299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:20.471349 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:20.471519 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:20.471520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:20.471675 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:20.471724 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.471725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.472362 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.472362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.472897 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.472899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.473215 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.473216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.473254 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.484026 (+ 10772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:20.484033 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:20.486053 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260925 07:21:20.486308 13456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:20.471228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:37725 (local address 127.4.47.62:40921)
0925 07:21:20.471344 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:20.471346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:20.471356 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:20.471396 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:20.471397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:20.471407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:20.471478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:20.471793 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.472294 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.472967 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.473152 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.473204 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.486184 (+ 12980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:37725: BlockingRecv error: recv got EOF from 127.4.47.1:37725 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:20.591018 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:20.586276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:43255)
0925 07:21:20.586402 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:20.586416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:20.586471 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:20.586600 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:20.586600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:20.586776 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:20.586823 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.586824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.587563 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.587564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.588145 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.588147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.588568 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.588569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.588620 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.589113 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:20.589118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:20.590956 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
W20260925 07:21:20.591161 13456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:20.586343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:43255 (local address 127.4.47.62:40921)
0925 07:21:20.586441 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:20.586443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:20.586511 (+ 68us) server_negotiation.cc:407] Connection header received
0925 07:21:20.586537 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:20.586538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:20.586548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:20.586591 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:20.586906 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.587485 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.588236 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.588431 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.588513 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.591053 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:43255: BlockingRecv error: recv got EOF from 127.4.47.2:43255 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:20 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:20.728940 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:20.723975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:44315)
0925 07:21:20.724168 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:20.724180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:20.724259 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:20.724412 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:20.724413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:20.724592 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:20.724639 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.724640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.725425 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.725427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.726120 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:20.726123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.726498 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.726499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:20.726555 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.727056 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:20.727087 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:20.728882 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
W20260925 07:21:20.729125 13456 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:20.724047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:44315 (local address 127.4.47.62:40921)
0925 07:21:20.724123 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:20.724125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:20.724241 (+ 116us) server_negotiation.cc:407] Connection header received
0925 07:21:20.724314 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:20.724316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:20.724338 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:20.724404 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:20.724724 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.725349 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.726206 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:20.726413 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:20.726500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:20.729049 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:44315: BlockingRecv error: recv got EOF from 127.4.47.3:44315 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:21.491364 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:21.486679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:60867)
0925 07:21:21.486829 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:21.486841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:21.486927 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:21.487111 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:21.487112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:21.487293 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:21.487338 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.487339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.488070 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.488071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.488607 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.488608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.488994 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.488995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.489052 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.489493 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:21.489498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:21.491263 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
W20260925 07:21:21.491472 13460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:21.486711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:60867 (local address 127.4.47.62:40921)
0925 07:21:21.486834 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:21.486836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:21.486919 (+ 83us) server_negotiation.cc:407] Connection header received
0925 07:21:21.486984 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:21.486985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:21.486998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:21.487107 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:21.487469 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.487966 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.488693 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.488903 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.488994 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.491355 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:60867: BlockingRecv error: recv got EOF from 127.4.47.1:60867 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:21.596966 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:21.591594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:56295)
0925 07:21:21.591731 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:21.591746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:21.591812 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:21.592020 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:21.592021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:21.592198 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:21.592264 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.592266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.593024 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.593025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.593591 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.593593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.594152 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.594153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.594235 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.594687 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:21.594691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:21.596855 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260925 07:21:21.597059 13460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:21.591711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56295 (local address 127.4.47.62:40921)
0925 07:21:21.591778 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:21.591780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:21.591793 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:21:21.591903 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:21.591903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:21.591916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:21.592009 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:21.592347 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.592926 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.593679 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.594047 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.594154 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.596947 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:56295: BlockingRecv error: recv got EOF from 127.4.47.2:56295 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:21 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:21.733824 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:21.729419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:42773)
0925 07:21:21.729579 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:21.729598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:21.729654 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:21.729765 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:21.729766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:21.729922 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:21.729973 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.729975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.730615 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.730615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.731153 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:21.731154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.731499 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.731500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:21.731539 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.731970 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:21.731975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:21.733761 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
W20260925 07:21:21.733928 13460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:21.729489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:42773 (local address 127.4.47.62:40921)
0925 07:21:21.729574 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:21.729577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:21.729655 (+ 78us) server_negotiation.cc:407] Connection header received
0925 07:21:21.729698 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:21.729699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:21.729711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:21.729762 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:21.730032 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.730544 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.731220 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:21.731397 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:21.731450 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:21.733860 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:42773: BlockingRecv error: recv got EOF from 127.4.47.3:42773 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:22.496595 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:22.491854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:43351)
0925 07:21:22.492016 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:22.492028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:22.492083 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:22.492253 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:22.492254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:22.492421 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:22.492469 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.492471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.493189 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.493190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.493714 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.493715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.494094 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.494095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.494153 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.494591 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:22.494595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:22.496514 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260925 07:21:22.496759 13464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:22.491940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:43351 (local address 127.4.47.62:40921)
0925 07:21:22.492050 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:22.492052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:22.492065 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:21:22.492158 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:22.492159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:22.492170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:22.492234 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:22.492590 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.493114 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.493796 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.494004 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.494063 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.496665 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:43351: BlockingRecv error: recv got EOF from 127.4.47.1:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:22.602442 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:22.597500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:54451)
0925 07:21:22.597661 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:22.597673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:22.597739 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:22.597875 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:22.597876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:22.598044 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:22.598089 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.598090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.598891 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.598891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.599528 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.599531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.599954 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.599955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.600019 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.600542 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:22.600547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:22.602373 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
W20260925 07:21:22.602605 13464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:22.597572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:54451 (local address 127.4.47.62:40921)
0925 07:21:22.597648 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:22.597651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:22.597740 (+ 89us) server_negotiation.cc:407] Connection header received
0925 07:21:22.597789 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:22.597790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:22.597803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:22.597854 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:22.598159 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.598813 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.599606 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.599874 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.599948 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.602504 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:54451: BlockingRecv error: recv got EOF from 127.4.47.2:54451 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:22 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:22.739001 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:22.734323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:34339)
0925 07:21:22.734493 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:22.734507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:22.734577 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:22.734728 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:22.734729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:22.734928 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:22.734974 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.734976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.735712 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.735713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.736248 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:22.736250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.736619 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.736620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:22.736677 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.737120 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:22.737124 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:22.738943 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260925 07:21:22.739174 13464 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:22.734411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:34339 (local address 127.4.47.62:40921)
0925 07:21:22.734497 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:22.734500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:22.734559 (+ 59us) server_negotiation.cc:407] Connection header received
0925 07:21:22.734631 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:22.734632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:22.734646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:22.734717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:22.735058 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.735622 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.736335 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:22.736531 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:22.736617 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:22.739067 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:34339: BlockingRecv error: recv got EOF from 127.4.47.3:34339 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:23.501879 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:23.497050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:46653)
0925 07:21:23.497194 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:23.497204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:23.497258 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:23.497472 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:23.497472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:23.497631 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:23.497680 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.497681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.498489 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.498491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.499029 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.499031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.499482 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.499483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.499526 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.499957 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:23.499979 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:23.501820 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
W20260925 07:21:23.502053 13468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:23.497131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46653 (local address 127.4.47.62:40921)
0925 07:21:23.497236 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:23.497238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:23.497248 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:23.497360 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:23.497361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:23.497374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:23.497451 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:23.497748 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.498406 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.499182 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.499377 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.499475 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.501963 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:46653: BlockingRecv error: recv got EOF from 127.4.47.1:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:23.607738 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:23.603035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:42957)
0925 07:21:23.603217 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:23.603229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:23.603320 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:23.603479 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:23.603479 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:23.603728 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:23.603787 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.603790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.604493 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.604494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.605080 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.605082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.605389 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.605389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.605427 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.605849 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:23.605854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:23.607680 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
W20260925 07:21:23.607844 13468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:23.603156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:42957 (local address 127.4.47.62:40921)
0925 07:21:23.603211 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:23.603214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:23.603359 (+ 145us) server_negotiation.cc:407] Connection header received
0925 07:21:23.603383 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:23.603384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:23.603394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:23.603471 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:23.603862 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.604424 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.605150 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.605327 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.605377 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.607773 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:42957: BlockingRecv error: recv got EOF from 127.4.47.2:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:23 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:23.744004 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:23.739576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:38151)
0925 07:21:23.739721 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:23.739739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:23.739800 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:23.739942 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:23.739943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:23.740106 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:23.740156 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.740157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.740792 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.740793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.741318 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:23.741321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.741663 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.741663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:23.741703 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.742144 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:23.742148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:23.743944 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260925 07:21:23.744204 13468 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:23.739649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:38151 (local address 127.4.47.62:40921)
0925 07:21:23.739721 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:23.739723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:23.739808 (+ 85us) server_negotiation.cc:407] Connection header received
0925 07:21:23.739847 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:23.739848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:23.739861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:23.739915 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:23.740221 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.740701 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.741409 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:23.741602 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:23.741650 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:23.744067 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:38151: BlockingRecv error: recv got EOF from 127.4.47.3:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:24.506851 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:24.502329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:45183)
0925 07:21:24.502489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:24.502500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:24.502557 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:24.502687 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:24.502688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:24.502848 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:24.502896 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.502898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.503559 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.503559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.504133 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.504135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.504449 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.504450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.504495 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.504967 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:24.504971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:24.506756 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
W20260925 07:21:24.506906 13472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:24.502427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:45183 (local address 127.4.47.62:40921)
0925 07:21:24.502559 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:24.502562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:24.502587 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:21:24.502608 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:24.502609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:24.502620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:24.502670 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:24.502959 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.503490 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.504203 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.504389 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.504438 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.506834 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:45183: BlockingRecv error: recv got EOF from 127.4.47.1:45183 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:24.613040 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:24.608236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:49023)
0925 07:21:24.608382 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:24.608392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:24.608437 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:24.608573 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:24.608574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:24.608733 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:24.608779 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.608780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.609420 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.609421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.610004 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.610005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.610392 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.610393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.610433 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.610979 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:24.610983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:24.612961 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260925 07:21:24.613155 13472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:24.608349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:49023 (local address 127.4.47.62:40921)
0925 07:21:24.608414 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:24.608418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:24.608431 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:21:24.608496 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:24.608496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:24.608508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:24.608554 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:24.608842 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.609339 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.610104 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.610317 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.610385 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.613061 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:49023: BlockingRecv error: recv got EOF from 127.4.47.2:49023 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:24 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:24.749294 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:24.744561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:42237)
0925 07:21:24.744730 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:24.744742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:24.744824 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:24.744970 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:24.744971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:24.745168 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:24.745210 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.745212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.745904 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.745905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.746445 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:24.746446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.746825 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.746826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:24.746891 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.747379 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:24.747383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:24.749238 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
W20260925 07:21:24.749396 13472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:24.744743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:42237 (local address 127.4.47.62:40921)
0925 07:21:24.744812 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:24.744814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:24.744824 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:24.744882 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:24.744883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:24.744898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:24.744955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:24.745298 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.745818 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.746550 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:24.746742 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:24.746824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:24.749326 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:42237: BlockingRecv error: recv got EOF from 127.4.47.3:42237 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:25.518008 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:25.507399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:36411)
0925 07:21:25.507561 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:25.507572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:25.507624 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:25.507786 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:25.507787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:25.507951 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:25.508002 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.508004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.508701 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.508702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.509225 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.509226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.509638 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.509639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.509691 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.516154 (+ 6463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:25.516159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:25.517950 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
W20260925 07:21:25.518178 13476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:25.507507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:36411 (local address 127.4.47.62:40921)
0925 07:21:25.507646 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:25.507648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:25.507664 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:21:25.507689 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:25.507689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:25.507700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:25.507773 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:25.508079 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.508630 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.509305 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.509542 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.509599 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.518096 (+ 8497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:36411: BlockingRecv error: recv got EOF from 127.4.47.1:36411 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:25.618929 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:25.613747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:54429)
0925 07:21:25.613982 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:25.613994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:25.614101 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:25.614280 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:25.614281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:25.614530 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:25.614577 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.614580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.615324 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.615326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.615861 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.615863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.616237 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.616238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.616296 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.616866 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:25.616871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:25.618847 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
W20260925 07:21:25.619109 13476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:25.613864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:54429 (local address 127.4.47.62:40921)
0925 07:21:25.613965 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:25.613968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:25.614081 (+ 113us) server_negotiation.cc:407] Connection header received
0925 07:21:25.614180 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:25.614181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:25.614196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:25.614272 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:25.614668 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.615241 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.615954 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.616152 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.616237 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.619000 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:54429: BlockingRecv error: recv got EOF from 127.4.47.2:54429 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:25 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:25.754814 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:25.749869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:55581)
0925 07:21:25.750050 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:25.750061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:25.750123 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:25.750256 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:25.750257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:25.750415 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:25.750462 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.750464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.751227 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.751227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.751813 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:25.751815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.752210 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.752210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:25.752251 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.752806 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:25.752810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:25.754757 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
W20260925 07:21:25.754927 13476 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:25.749936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:55581 (local address 127.4.47.62:40921)
0925 07:21:25.750017 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:25.750019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:25.750126 (+ 107us) server_negotiation.cc:407] Connection header received
0925 07:21:25.750179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:25.750180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:25.750193 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:25.750235 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:25.750560 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.751159 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.751891 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:25.752140 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:25.752199 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:25.754850 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:55581: BlockingRecv error: recv got EOF from 127.4.47.3:55581 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:26.523334 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:26.518442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:39077)
0925 07:21:26.518622 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:26.518637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:26.518694 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:26.518824 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:26.518825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:26.518988 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:26.519056 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.519057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.519805 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.519806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.520350 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.520352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.520768 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.520769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.520811 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.521337 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:26.521342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:26.523250 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
W20260925 07:21:26.523425 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:26.518506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:39077 (local address 127.4.47.62:40921)
0925 07:21:26.518632 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:26.518634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:26.518679 (+ 45us) server_negotiation.cc:407] Connection header received
0925 07:21:26.518737 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:26.518738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:26.518748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:26.518806 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:26.519191 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.519721 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.520481 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.520690 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.520754 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.523347 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:39077: BlockingRecv error: recv got EOF from 127.4.47.1:39077 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:26.624074 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:26.619489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:47755)
0925 07:21:26.619664 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:26.619682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:26.619738 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:26.619866 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:26.619867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:26.620031 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:26.620079 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.620081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.620745 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.620745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.621293 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.621295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.621639 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.621640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.621683 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.622121 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:26.622126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:26.623993 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":123,"threads_started":1}
W20260925 07:21:26.624153 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:26.619576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:47755 (local address 127.4.47.62:40921)
0925 07:21:26.619653 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:26.619654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:26.619737 (+ 83us) server_negotiation.cc:407] Connection header received
0925 07:21:26.619783 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:26.619784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:26.619794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:26.619847 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:26.620160 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.620673 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.621367 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.621576 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.621629 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.624106 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:47755: BlockingRecv error: recv got EOF from 127.4.47.2:47755 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:26 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:26.760090 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:26.755336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:50673)
0925 07:21:26.755550 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:26.755562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:26.755631 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:26.755774 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:26.755774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:26.755932 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:26.755991 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.755992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.756628 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.756628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.757154 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:26.757156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.757534 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.757535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:26.757572 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.758086 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:26.758091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:26.760012 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
W20260925 07:21:26.760231 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:26.755411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:50673 (local address 127.4.47.62:40921)
0925 07:21:26.755488 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:26.755490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:26.755634 (+ 144us) server_negotiation.cc:407] Connection header received
0925 07:21:26.755694 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:26.755695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:26.755708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:26.755751 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:26.756054 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.756556 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.757254 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:26.757456 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:26.757523 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:26.760132 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:50673: BlockingRecv error: recv got EOF from 127.4.47.3:50673 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:27.528616 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:27.523910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:42693)
0925 07:21:27.524068 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:27.524084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:27.524137 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:27.524297 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:27.524298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:27.524464 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:27.524511 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.524513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.525174 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.525175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.525820 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.525823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.526157 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.526158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.526200 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.526683 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:27.526688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:27.528548 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260925 07:21:27.528702 13484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:27.523974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:42693 (local address 127.4.47.62:40921)
0925 07:21:27.524137 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:27.524139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:27.524187 (+ 48us) server_negotiation.cc:407] Connection header received
0925 07:21:27.524210 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:27.524211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:27.524222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:27.524274 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:27.524584 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.525106 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.525896 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.526091 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.526144 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.528647 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:42693: BlockingRecv error: recv got EOF from 127.4.47.1:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":112,"threads_started":1}
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:27.629551 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:27.624585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:59297)
0925 07:21:27.624727 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:27.624739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:27.624800 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:27.625015 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:27.625016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:27.625189 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:27.625236 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.625238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.625929 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.625930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.626518 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.626520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.626900 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.626901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.626955 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.627432 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:27.627437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:27.629494 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260925 07:21:27.629695 13484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:27.624743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:59297 (local address 127.4.47.62:40921)
0925 07:21:27.624798 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:27.624800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:27.624811 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:21:27.624865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:27.624866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:27.624884 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:27.625006 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:27.625323 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.625838 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.626616 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.626814 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.626901 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.629616 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:59297: BlockingRecv error: recv got EOF from 127.4.47.2:59297 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:27 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:27.765185 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:27.760611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:47733)
0925 07:21:27.760773 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:27.760784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:27.760849 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:27.760956 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:27.760957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:27.761115 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:27.761175 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.761177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.761814 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.761815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.762337 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:27.762339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.762661 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.762662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:27.762702 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.763257 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:27.763262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:27.765108 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
W20260925 07:21:27.765293 13484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:27.760669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:47733 (local address 127.4.47.62:40921)
0925 07:21:27.760735 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:27.760737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:27.760846 (+ 109us) server_negotiation.cc:407] Connection header received
0925 07:21:27.760891 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:27.760891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:27.760903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:27.760952 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:27.761232 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.761738 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.762407 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:27.762592 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:27.762646 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:27.765197 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:47733: BlockingRecv error: recv got EOF from 127.4.47.3:47733 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:28.533849 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:28.529154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:60621)
0925 07:21:28.529287 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:28.529299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:28.529371 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:28.529523 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:28.529523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:28.529693 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:28.529743 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.529744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.530406 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.530407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.530987 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.530990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.531341 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.531342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.531392 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.531839 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:28.531844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:28.533779 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
W20260925 07:21:28.533967 13488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:28.529243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:60621 (local address 127.4.47.62:40921)
0925 07:21:28.529388 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:28.529390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:28.529398 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:21:28.529421 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:28.529421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:28.529432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:28.529509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:28.529812 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.530328 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.531081 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.531273 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.531324 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.533888 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:60621: BlockingRecv error: recv got EOF from 127.4.47.1:60621 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:28.634704 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:28.630030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:44585)
0925 07:21:28.630202 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:28.630212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:28.630270 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:28.630407 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:28.630408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:28.630568 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:28.630619 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.630621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.631322 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.631322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.631835 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.631837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.632167 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.632168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.632205 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.632729 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:28.632734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:28.634645 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
W20260925 07:21:28.634778 13488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:28.630127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:44585 (local address 127.4.47.62:40921)
0925 07:21:28.630226 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:28.630229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:28.630264 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:21:28.630317 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:28.630318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:28.630329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:28.630388 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:28.630687 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.631252 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.631905 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.632094 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.632145 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.634730 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:44585: BlockingRecv error: recv got EOF from 127.4.47.2:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:28 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.3@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:28.769991 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:28.765626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.3:56283)
0925 07:21:28.765744 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:28.765754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:28.765811 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:28.765919 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:28.765919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:28.766095 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:28.766174 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.766176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.766849 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.766850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.767408 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:28.767410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.767716 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.767716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:28.767754 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.768249 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:28.768254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:28.769934 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
W20260925 07:21:28.770269 13488 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:28.765696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:56283 (local address 127.4.47.62:40921)
0925 07:21:28.765769 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:28.765771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:28.765797 (+ 26us) server_negotiation.cc:407] Connection header received
0925 07:21:28.765856 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:28.765857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:28.765869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:28.765915 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:28.766291 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.766779 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.767474 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:28.767652 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:28.767703 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:28.770188 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.3:56283: BlockingRecv error: recv got EOF from 127.4.47.3:56283 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 07:21:29 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:29 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.1@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:29.539261 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:29.534460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.1:52683)
0925 07:21:29.534660 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:29.534672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:29.534755 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:29.534906 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:29.534907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:29.535077 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:29.535126 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:29.535128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.535784 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.535784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:29.536299 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:29.536300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.536669 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.536670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:29.536733 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:29.537250 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:29.537255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:29.539157 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
W20260925 07:21:29.539289 13492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:29.534562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52683 (local address 127.4.47.62:40921)
0925 07:21:29.534715 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:29.534717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:29.534736 (+ 19us) server_negotiation.cc:407] Connection header received
0925 07:21:29.534815 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:29.534816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:29.534828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:29.534898 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:29.535198 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.535710 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.536376 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.536586 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.536642 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:29.539219 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.1:52683: BlockingRecv error: recv got EOF from 127.4.47.1:52683 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
Sep 25 07:21:29 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:21:29 dist-test-slave-zw07 krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.47.2@KRBTEST.COM for oryx/127.4.47.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:21:29.641178 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:29.635822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40921 (local address 127.4.47.2:58615)
0925 07:21:29.635995 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:29.636005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:29.636061 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:29.636264 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:29.636266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:29.636531 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:29.636592 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:29.636594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.637270 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.637271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:29.637869 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:29.637871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.638283 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.638285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:29.638356 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:29.638926 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:29.638931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:29.641082 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.47.62:40921: Server oryx/127.4.47.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260925 07:21:29.641263 13492 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:29.635892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:58615 (local address 127.4.47.62:40921)
0925 07:21:29.635964 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:29.635967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:29.636084 (+ 117us) server_negotiation.cc:407] Connection header received
0925 07:21:29.636128 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:29.636131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:29.636154 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:29.636208 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:29.636665 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.637195 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.637950 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.638188 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.638276 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:29.641198 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.47.2:58615: BlockingRecv error: recv got EOF from 127.4.47.2:58615 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:21:29.690088 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13018
I20260925 07:21:29.694679 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13169
I20260925 07:21:29.699770 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13301
I20260925 07:21:29.704075 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 12947
2026-09-25T07:21:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17420 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T07:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:29Z Disabled control of system clock
I20260925 07:21:29.731602 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40563
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:40563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 07:21:29.811520 13497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:29.811735 13497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:29.811755 13497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:29.812947 13497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:29.812991 13497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:29.813004 13497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:29.813058 13497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:29.813084 13497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:29.814409 13497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:40563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:29.814694 13497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:29.815006 13497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:29.820453 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:29.820489 13497 server_base.cc:1034] running on GCE node
W20260925 07:21:29.820528 13502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:29.820575 13503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:29.820976 13497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:29.821240 13497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:29.822384 13497 hybrid_clock.cc:648] HybridClock initialized: now 1790320889822374 us; error 33 us; skew 500 ppm
I20260925 07:21:29.827741 13497 webserver.cc:460] Webserver started at http://127.4.47.62:37995/ using document root <none> and password file <none>
I20260925 07:21:29.828002 13497 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:29.828053 13497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:29.828223 13497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:29.829342 13497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "0fc1b5827e70419aa7f608be614aa29c"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "cf7b7e6a93c764c637e72b40455e9508"
server_key_iv: "735235397d66ce281b6440b69776da8c"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.829730 13497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "0fc1b5827e70419aa7f608be614aa29c"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "cf7b7e6a93c764c637e72b40455e9508"
server_key_iv: "735235397d66ce281b6440b69776da8c"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.831051 13497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:21:29.831889 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:29.832077 13497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:21:29.832175 13497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "0fc1b5827e70419aa7f608be614aa29c"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "cf7b7e6a93c764c637e72b40455e9508"
server_key_iv: "735235397d66ce281b6440b69776da8c"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.832253 13497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:29.845230 13497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:29.845572 13497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:29.845742 13497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:29.849809 13497 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:40563
I20260925 07:21:29.849840 13563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:40563 every 8 connection(s)
I20260925 07:21:29.850196 13497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:29.850880 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:29.853477 13564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c: Bootstrap starting.
I20260925 07:21:29.854077 13564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:29.854383 13564 log.cc:824] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:29.855042 13564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c: No bootstrap required, opened a new log
I20260925 07:21:29.855933 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13497
I20260925 07:21:29.856014 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:29.856205 4284 external_mini_cluster.cc:1428] Setting key e5515440b9ed4eec1dcd016a6f74bf22
I20260925 07:21:29.856932 13564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } }
I20260925 07:21:29.857115 13564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:29.857161 13564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc1b5827e70419aa7f608be614aa29c, State: Initialized, Role: FOLLOWER
I20260925 07:21:29.857290 13564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } }
I20260925 07:21:29.857388 13564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:29.857422 13564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:29.857471 13564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:29.857991 13564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } }
I20260925 07:21:29.858079 13564 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc1b5827e70419aa7f608be614aa29c; no voters:
I20260925 07:21:29.858255 13564 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:29.858292 13569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:29.858410 13569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [term 1 LEADER]: Becoming Leader. State: Replica: 0fc1b5827e70419aa7f608be614aa29c, State: Running, Role: LEADER
I20260925 07:21:29.858505 13564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:29.858487 13569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } }
I20260925 07:21:29.858796 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc1b5827e70419aa7f608be614aa29c. Latest consensus state: current_term: 1 leader_uuid: "0fc1b5827e70419aa7f608be614aa29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } } }
I20260925 07:21:29.858871 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:29.858807 13571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc1b5827e70419aa7f608be614aa29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1b5827e70419aa7f608be614aa29c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40563 } } }
I20260925 07:21:29.858975 13571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:29.859927 13574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:29.860456 13574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:29.862830 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:29.856801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.4.47.62:40563)
0925 07:21:29.857014 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:29.857021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:29.857045 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:21:29.857169 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:29.857176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:29.857256 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:29.857397 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:29.857592 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.859395 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.860529 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:29.861127 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:29.861248 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:29.862073 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:29.862076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:29.862077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:29.862289 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:29.862329 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:29.862344 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:29.862414 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:29.862507 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 07:21:29.862593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260925 07:21:29.863095 13574 catalog_manager.cc:1347] Generated new cluster ID: f5aa81c4e8fb48b7a15aa536f74a7c42
I20260925 07:21:29.863166 13574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:29.872678 13574 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:29.873220 13574 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:29.877568 13574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc1b5827e70419aa7f608be614aa29c: Generated new TSK 0
I20260925 07:21:29.877775 13574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:21:29.880007 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--builtin_ntp_servers=127.4.47.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 07:21:29.962297 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:29.962545 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:29.962584 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:29.963822 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:29.963888 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:29.963927 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:29.965251 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:29.965521 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:29.965821 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:29.970749 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:29.970820 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:29.970888 13588 server_base.cc:1034] running on GCE node
W20260925 07:21:29.970803 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:29.971428 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:29.971690 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:29.972867 13588 hybrid_clock.cc:648] HybridClock initialized: now 1790320889972840 us; error 39 us; skew 500 ppm
I20260925 07:21:29.974265 13588 webserver.cc:460] Webserver started at http://127.4.47.1:43799/ using document root <none> and password file <none>
I20260925 07:21:29.974534 13588 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:29.974593 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:29.974833 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:29.976305 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "4a397c668b03410ba227b4f4859a70e1"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "3a95ee9817e4416d3573e282bbb67825"
server_key_iv: "2a5f4882e8d948d7bd8484878c60a7ce"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.976699 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "4a397c668b03410ba227b4f4859a70e1"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "3a95ee9817e4416d3573e282bbb67825"
server_key_iv: "2a5f4882e8d948d7bd8484878c60a7ce"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.978367 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:29.979387 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:29.979563 13588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:29.979672 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "4a397c668b03410ba227b4f4859a70e1"
format_stamp: "Formatted at 2026-09-25 07:21:29 on dist-test-slave-zw07"
server_key: "3a95ee9817e4416d3573e282bbb67825"
server_key_iv: "2a5f4882e8d948d7bd8484878c60a7ce"
server_key_version: "encryptionkey@0"
I20260925 07:21:29.979749 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:29.998253 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:29.998587 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:29.998744 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:29.998987 13588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:29.999356 13588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:29.999392 13588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:29.999414 13588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:29.999466 13588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.006714 13588 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:38003
I20260925 07:21:30.006783 13715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:38003 every 8 connection(s)
I20260925 07:21:30.007323 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:21:30.014966 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.008178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:35007 (local address 127.4.47.62:40563)
0925 07:21:30.008229 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.008231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.008584 (+ 353us) server_negotiation.cc:407] Connection header received
0925 07:21:30.009589 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.009591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.009607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.009694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.010615 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.011272 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.013023 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.013345 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.013426 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.014676 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.014678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.014678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.014683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.014702 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.014707 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.014759 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.014868 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.014898 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:21:30.015393 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13588
I20260925 07:21:30.015457 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.008194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40563 (local address 127.4.47.1:35007)
0925 07:21:30.008436 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.008498 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.009231 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.009783 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.009789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.009803 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.010518 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.010526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.011374 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.011375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.012897 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.012901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.013500 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.013501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.013857 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.014495 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.014496 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.014543 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.014576 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.014776 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.014782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.014786 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.014867 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.015015 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260925 07:21:30.015496 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:30.015709 4284 external_mini_cluster.cc:1428] Setting key 10bfc4b23dce6b471f59c8a8919c520f
I20260925 07:21:30.016083 13716 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40563
I20260925 07:21:30.016249 13716 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.016526 13716 heartbeater.cc:511] Master 127.4.47.62:40563 requested a full tablet report, sending...
I20260925 07:21:30.016923 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--builtin_ntp_servers=127.4.47.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 07:21:30.017243 13528 ts_manager.cc:194] Registered new tserver with Master: 4a397c668b03410ba227b4f4859a70e1 (127.4.47.1:38003)
I20260925 07:21:30.018308 13528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.1:35007
W20260925 07:21:30.105842 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.106060 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.106081 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.107239 13719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.107275 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.107300 13719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:30.108592 13719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.108788 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.109045 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.114545 13725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.114562 13724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.114566 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.114825 13719 server_base.cc:1034] running on GCE node
I20260925 07:21:30.115044 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.115310 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.116478 13719 hybrid_clock.cc:648] HybridClock initialized: now 1790320890116469 us; error 31 us; skew 500 ppm
I20260925 07:21:30.117823 13719 webserver.cc:460] Webserver started at http://127.4.47.2:43031/ using document root <none> and password file <none>
I20260925 07:21:30.118103 13719 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.118180 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.118324 13719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.119467 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "7715da74380749e39e11b72d689e8181"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "1757f4cc7dd798ee503e0692c1cbaabe"
server_key_iv: "10b99e19094725c3d1f686e72aacec7c"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.119812 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "7715da74380749e39e11b72d689e8181"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "1757f4cc7dd798ee503e0692c1cbaabe"
server_key_iv: "10b99e19094725c3d1f686e72aacec7c"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.121084 13719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:21:30.121801 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.121953 13719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:30.122051 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "7715da74380749e39e11b72d689e8181"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "1757f4cc7dd798ee503e0692c1cbaabe"
server_key_iv: "10b99e19094725c3d1f686e72aacec7c"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.122121 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.133039 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.133344 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.133507 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.133721 13719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:30.134075 13719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:30.134163 13719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.134205 13719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:30.134227 13719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.139875 13719 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:40403
I20260925 07:21:30.139910 13846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:40403 every 8 connection(s)
I20260925 07:21:30.140282 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:30.142799 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13719
I20260925 07:21:30.142894 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:30.143121 4284 external_mini_cluster.cc:1428] Setting key 3d7ddee657fdb2c47a142cb8ebe18094
I20260925 07:21:30.144096 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--builtin_ntp_servers=127.4.47.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 07:21:30.145886 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.141094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:46785 (local address 127.4.47.62:40563)
0925 07:21:30.141170 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.141172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.141429 (+ 257us) server_negotiation.cc:407] Connection header received
0925 07:21:30.141892 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.141893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.141905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.141959 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.142611 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.143281 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.144444 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.144667 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.144747 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.145599 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.145601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.145601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.145606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.145621 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.145625 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.145681 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.145778 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.145807 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:21:30.146112 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.141086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40563 (local address 127.4.47.2:46785)
0925 07:21:30.141295 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.141327 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.141803 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.142034 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.142037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.142046 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.142533 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.142537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.143372 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.143374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.144353 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.144356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.144752 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.144754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.144889 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.145454 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.145455 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.145482 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.145505 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.145682 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.145687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.145690 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.145779 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.145896 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260925 07:21:30.146673 13847 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40563
I20260925 07:21:30.146782 13847 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.146963 13847 heartbeater.cc:511] Master 127.4.47.62:40563 requested a full tablet report, sending...
I20260925 07:21:30.147403 13528 ts_manager.cc:194] Registered new tserver with Master: 7715da74380749e39e11b72d689e8181 (127.4.47.2:40403)
I20260925 07:21:30.147864 13528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.2:46785
W20260925 07:21:30.224645 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.224833 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.224854 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.226060 13850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.226097 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.226122 13850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:30.227533 13850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.227770 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.228083 13850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.233119 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.233174 13850 server_base.cc:1034] running on GCE node
W20260925 07:21:30.233178 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.233124 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.233510 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.233747 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.234923 13850 hybrid_clock.cc:648] HybridClock initialized: now 1790320890234910 us; error 44 us; skew 500 ppm
I20260925 07:21:30.236342 13850 webserver.cc:460] Webserver started at http://127.4.47.3:37501/ using document root <none> and password file <none>
I20260925 07:21:30.236598 13850 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.236666 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.236815 13850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.237870 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "fab36f47993e49dca60bd1229c4e3398"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "e2ad0499c4818e6e98a19d7f92883c63"
server_key_iv: "c1cd4b38ac84cf8bc48663305b3affa0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.238226 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "fab36f47993e49dca60bd1229c4e3398"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "e2ad0499c4818e6e98a19d7f92883c63"
server_key_iv: "c1cd4b38ac84cf8bc48663305b3affa0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.239631 13850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:21:30.240334 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.240502 13850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.240599 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "fab36f47993e49dca60bd1229c4e3398"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "e2ad0499c4818e6e98a19d7f92883c63"
server_key_iv: "c1cd4b38ac84cf8bc48663305b3affa0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.240682 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.277156 13850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.277444 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.277549 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.277740 13850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:30.278090 13850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:30.278123 13850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.278177 13850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:30.278211 13850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.284197 13850 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:45365
I20260925 07:21:30.284240 13977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:45365 every 8 connection(s)
I20260925 07:21:30.284564 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:30.289317 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13850
I20260925 07:21:30.289417 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:30.289587 4284 external_mini_cluster.cc:1428] Setting key c8872eb3eeaba444b28bb755b8a21649
I20260925 07:21:30.289750 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.285322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:36281 (local address 127.4.47.62:40563)
0925 07:21:30.285383 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.285385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.285613 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:21:30.286109 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.286110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.286120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.286170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.286799 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.287367 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.288316 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.288491 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.288539 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.289357 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.289358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.289358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.289362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.289377 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.289382 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.289468 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.289630 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.289668 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:30.289978 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.285306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40563 (local address 127.4.47.3:36281)
0925 07:21:30.285483 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.285527 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.285999 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.286226 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.286229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.286237 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.286724 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.286729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.287446 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.287446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.288236 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:30.288238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:30.288562 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:30.288563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:30.288667 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:30.289190 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.289190 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.289232 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.289261 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.289510 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.289528 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.289531 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.289631 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.289732 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260925 07:21:30.290442 13978 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40563
I20260925 07:21:30.290539 13978 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.290784 13978 heartbeater.cc:511] Master 127.4.47.62:40563 requested a full tablet report, sending...
I20260925 07:21:30.291215 13528 ts_manager.cc:194] Registered new tserver with Master: fab36f47993e49dca60bd1229c4e3398 (127.4.47.3:45365)
I20260925 07:21:30.291595 13528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.3:36281
I20260925 07:21:30.301674 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:21:30.303540 13568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:30.303097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.4.47.62:40563)
0925 07:21:30.303160 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.303162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.303252 (+ 90us) server_negotiation.cc:407] Connection header received
0925 07:21:30.303308 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.303308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.303317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.303358 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.303488 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39446: BlockingRecv error: recv got EOF from 127.0.0.1:39446 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:30.303618 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13588
I20260925 07:21:30.307965 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13719
I20260925 07:21:30.312376 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13850
I20260925 07:21:30.317088 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13497
2026-09-25T07:21:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (611 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T07:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:30Z Disabled control of system clock
I20260925 07:21:30.343583 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41417
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:43343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:21:30.424044 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.424257 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.424278 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.425494 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:30.425540 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.425554 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.425565 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:30.425577 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:30.426875 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:41417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.427136 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.427450 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.432374 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.432416 13989 server_base.cc:1034] running on GCE node
W20260925 07:21:30.432387 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.432382 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.432857 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.433108 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.434257 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790320890434237 us; error 38 us; skew 500 ppm
I20260925 07:21:30.435648 13989 webserver.cc:460] Webserver started at http://127.4.47.62:33863/ using document root <none> and password file <none>
I20260925 07:21:30.435916 13989 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.435966 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.436136 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.437232 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "fd9711aab5d945529406dcf6747eb8b1"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "26c0aa72943ef7a765f7c413f803371e"
server_key_iv: "9af359ebac0bd081dc061cc9e03de704"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.437592 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "fd9711aab5d945529406dcf6747eb8b1"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "26c0aa72943ef7a765f7c413f803371e"
server_key_iv: "9af359ebac0bd081dc061cc9e03de704"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.438874 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:30.439654 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.439827 13989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.439918 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "fd9711aab5d945529406dcf6747eb8b1"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "26c0aa72943ef7a765f7c413f803371e"
server_key_iv: "9af359ebac0bd081dc061cc9e03de704"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.439992 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.443475 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.443753 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.443934 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.447808 13989 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:41417
I20260925 07:21:30.447837 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:41417 every 8 connection(s)
I20260925 07:21:30.448158 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:30.448764 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:30.451457 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1: Bootstrap starting.
I20260925 07:21:30.451998 14056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:30.452286 14056 log.cc:824] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:30.452907 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1: No bootstrap required, opened a new log
I20260925 07:21:30.454438 14056 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } }
I20260925 07:21:30.454593 14056 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:30.454638 14056 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9711aab5d945529406dcf6747eb8b1, State: Initialized, Role: FOLLOWER
I20260925 07:21:30.454799 14056 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } }
I20260925 07:21:30.454896 14056 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:30.454939 14056 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:30.454993 14056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:30.455531 14056 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } }
I20260925 07:21:30.455647 14056 leader_election.cc:302] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9711aab5d945529406dcf6747eb8b1; no voters:
I20260925 07:21:30.455814 14056 leader_election.cc:288] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:30.455849 14059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:30.456012 14059 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [term 1 LEADER]: Becoming Leader. State: Replica: fd9711aab5d945529406dcf6747eb8b1, State: Running, Role: LEADER
I20260925 07:21:30.456061 14056 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:30.456099 14059 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } }
I20260925 07:21:30.456779 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9711aab5d945529406dcf6747eb8b1. Latest consensus state: current_term: 1 leader_uuid: "fd9711aab5d945529406dcf6747eb8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } } }
I20260925 07:21:30.456876 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:30.457041 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9711aab5d945529406dcf6747eb8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9711aab5d945529406dcf6747eb8b1" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41417 } } }
I20260925 07:21:30.457113 14069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:30.457197 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:30.457641 14069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:30.457752 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 13989
I20260925 07:21:30.457849 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:30.458032 4284 external_mini_cluster.cc:1428] Setting key 0cea8058be14dd8d4fddee39d2291d34
I20260925 07:21:30.459859 14069 catalog_manager.cc:1347] Generated new cluster ID: 7013b85d8e224b918a2e3401fca4bfd9
I20260925 07:21:30.459916 14069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:21:30.466462 14069 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:30.466702 14069 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:30.466938 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.458667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.4.47.62:41417)
0925 07:21:30.458842 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.458850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.458878 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:21:30.459000 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.459012 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.459092 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.459251 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.465644 (+ 6393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.465684 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.465685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.465690 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.465741 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.465786 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.465947 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.466385 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.466449 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260925 07:21:30.471006 14069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9711aab5d945529406dcf6747eb8b1: Generated new TSK 0
I20260925 07:21:30.471239 14069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:21:30.475978 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--builtin_ntp_servers=127.4.47.20:43343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:21:30.557729 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.557933 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.557957 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.559173 14080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.559211 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.559237 14080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:30.560678 14080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.560904 14080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.561216 14080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.567031 14085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.567042 14086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.567046 14088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.567260 14080 server_base.cc:1034] running on GCE node
I20260925 07:21:30.567524 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.567793 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.568987 14080 hybrid_clock.cc:648] HybridClock initialized: now 1790320890568978 us; error 31 us; skew 500 ppm
I20260925 07:21:30.574066 14080 webserver.cc:460] Webserver started at http://127.4.47.1:39159/ using document root <none> and password file <none>
I20260925 07:21:30.574347 14080 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.574394 14080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.574538 14080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.575709 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "7ddf5062798548c3bf26c52293eaf3ac"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "6c90196e97d3ae549db111f9b7ee646e"
server_key_iv: "32c64d84bb2c6dd1f9af3f0c00d8f646"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.576076 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "7ddf5062798548c3bf26c52293eaf3ac"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "6c90196e97d3ae549db111f9b7ee646e"
server_key_iv: "32c64d84bb2c6dd1f9af3f0c00d8f646"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.577407 14080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:21:30.578204 14094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.578356 14080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:30.578466 14080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "7ddf5062798548c3bf26c52293eaf3ac"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "6c90196e97d3ae549db111f9b7ee646e"
server_key_iv: "32c64d84bb2c6dd1f9af3f0c00d8f646"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.578557 14080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.582082 14080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.582355 14080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.582505 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.582717 14080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:30.583017 14080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:30.583050 14080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.583127 14080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:30.583163 14080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.588882 14080 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:41875
I20260925 07:21:30.588927 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:41875 every 8 connection(s)
I20260925 07:21:30.589246 14080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:21:30.590919 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14080
I20260925 07:21:30.591005 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:30.591238 4284 external_mini_cluster.cc:1428] Setting key 46ba3344bdf9847eb79b3bd39dc44e44
I20260925 07:21:30.592339 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--builtin_ntp_servers=127.4.47.20:43343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 07:21:30.592623 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.589946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:36407 (local address 127.4.47.62:41417)
0925 07:21:30.590007 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.590010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.590230 (+ 220us) server_negotiation.cc:407] Connection header received
0925 07:21:30.590703 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.590704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.590711 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.590829 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.592367 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.592368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.592369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.592373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.592389 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.592394 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.592441 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.592539 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.592556 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:30.592829 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.589910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41417 (local address 127.4.47.1:36407)
0925 07:21:30.590100 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.590139 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.590607 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.590899 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.590903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.590911 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.592213 (+ 1302us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.592214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.592242 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.592272 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.592449 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.592453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.592456 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.592549 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.592611 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 07:21:30.593430 14208 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41417
I20260925 07:21:30.593547 14208 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.593724 14208 heartbeater.cc:511] Master 127.4.47.62:41417 requested a full tablet report, sending...
I20260925 07:21:30.594295 14020 ts_manager.cc:194] Registered new tserver with Master: 7ddf5062798548c3bf26c52293eaf3ac (127.4.47.1:41875)
W20260925 07:21:30.675457 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.675675 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.675697 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.676887 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.676928 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.676954 14211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:30.678308 14211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.678570 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.678864 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.684281 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.684302 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.684304 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.684576 14211 server_base.cc:1034] running on GCE node
I20260925 07:21:30.684787 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.685036 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.686194 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790320890686185 us; error 36 us; skew 500 ppm
I20260925 07:21:30.687475 14211 webserver.cc:460] Webserver started at http://127.4.47.2:39367/ using document root <none> and password file <none>
I20260925 07:21:30.687713 14211 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.687759 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.687932 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.689013 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "c37c7235188b4b4180a144f827786def"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "9c5a340cb927c4246a9eb3d0b9be4abe"
server_key_iv: "aa0c8d833d07f88e2e404803bd595dc0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.689378 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "c37c7235188b4b4180a144f827786def"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "9c5a340cb927c4246a9eb3d0b9be4abe"
server_key_iv: "aa0c8d833d07f88e2e404803bd595dc0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.690680 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:30.691501 14225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.691679 14211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.691784 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "c37c7235188b4b4180a144f827786def"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "9c5a340cb927c4246a9eb3d0b9be4abe"
server_key_iv: "aa0c8d833d07f88e2e404803bd595dc0"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.691859 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.695474 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.695751 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.695894 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.696107 14211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:30.696416 14211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:30.696470 14211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.696509 14211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:30.696537 14211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.703802 14211 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:46037
I20260925 07:21:30.703866 14338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:46037 every 8 connection(s)
I20260925 07:21:30.704241 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:21:30.707688 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14211
I20260925 07:21:30.707793 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:30.707890 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.705019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:49987 (local address 127.4.47.62:41417)
0925 07:21:30.706022 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.706024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.706035 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:21:30.706065 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.706066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.706071 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.706129 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.707578 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.707580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.707581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.707586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.707603 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.707609 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.707663 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.707785 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.707804 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971}
I20260925 07:21:30.708029 4284 external_mini_cluster.cc:1428] Setting key b6701e26930dee0e40b499fa93946094
I20260925 07:21:30.708117 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.704997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41417 (local address 127.4.47.2:49987)
0925 07:21:30.705208 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.705255 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.705909 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.706222 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.706227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.706237 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.707391 (+ 1154us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.707391 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.707430 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.707466 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.707690 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.707696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.707700 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.707781 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.707848 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260925 07:21:30.708575 14339 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41417
I20260925 07:21:30.708690 14339 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.708841 14339 heartbeater.cc:511] Master 127.4.47.62:41417 requested a full tablet report, sending...
I20260925 07:21:30.709220 14020 ts_manager.cc:194] Registered new tserver with Master: c37c7235188b4b4180a144f827786def (127.4.47.2:46037)
I20260925 07:21:30.709231 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--builtin_ntp_servers=127.4.47.20:43343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:21:30.790817 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:30.791064 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:30.791128 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:30.792325 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:30.792390 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:30.792441 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:30.794327 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:43343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:30.794646 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:30.794947 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:30.800034 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.800051 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:30.800061 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:30.800256 14342 server_base.cc:1034] running on GCE node
I20260925 07:21:30.800521 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:30.800794 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:30.801941 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790320890801921 us; error 36 us; skew 500 ppm
I20260925 07:21:30.803454 14342 webserver.cc:460] Webserver started at http://127.4.47.3:35145/ using document root <none> and password file <none>
I20260925 07:21:30.803741 14342 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:30.803788 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:30.803941 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:30.805083 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "c6d87455463a4aa6882d0ea2a82f9f89"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "c36399f2e67354da8110887aaae95598"
server_key_iv: "9098c056cff2b9ffc96288fd499b58c1"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.805466 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "c6d87455463a4aa6882d0ea2a82f9f89"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "c36399f2e67354da8110887aaae95598"
server_key_iv: "9098c056cff2b9ffc96288fd499b58c1"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.806802 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:21:30.807644 14356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.807832 14342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.807921 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "c6d87455463a4aa6882d0ea2a82f9f89"
format_stamp: "Formatted at 2026-09-25 07:21:30 on dist-test-slave-zw07"
server_key: "c36399f2e67354da8110887aaae95598"
server_key_iv: "9098c056cff2b9ffc96288fd499b58c1"
server_key_version: "encryptionkey@0"
I20260925 07:21:30.807996 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:30.811405 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:30.811678 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:30.811818 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:30.812031 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:30.812333 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:30.812366 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.812387 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:30.812434 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:30.817989 14342 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:33449
I20260925 07:21:30.818037 14469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:33449 every 8 connection(s)
I20260925 07:21:30.818377 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:21:30.821200 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.819116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:51167 (local address 127.4.47.62:41417)
0925 07:21:30.819172 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.819174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.819415 (+ 241us) server_negotiation.cc:407] Connection header received
0925 07:21:30.819849 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.819850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.819855 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.819909 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.821004 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:30.821005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:30.821006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:21:30.821010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:30.821024 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:21:30.821028 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:30.821071 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:30.821135 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 07:21:30.821148 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:21:30.821393 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:30.819102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41417 (local address 127.4.47.3:51167)
0925 07:21:30.819298 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:30.819330 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:30.819769 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:30.819960 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:30.819963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:30.819971 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:30.820863 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:21:30.820864 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:30.820890 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:21:30.820918 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:30.821075 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:30.821077 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:30.821079 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:21:30.821128 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:21:30.821173 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260925 07:21:30.821902 14470 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41417
I20260925 07:21:30.821995 14470 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:30.822137 14470 heartbeater.cc:511] Master 127.4.47.62:41417 requested a full tablet report, sending...
I20260925 07:21:30.822494 14020 ts_manager.cc:194] Registered new tserver with Master: c6d87455463a4aa6882d0ea2a82f9f89 (127.4.47.3:33449)
I20260925 07:21:30.823741 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14342
I20260925 07:21:30.823812 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:30.824007 4284 external_mini_cluster.cc:1428] Setting key e949b3d8cc597ef0ab3aa25080c37fb2
I20260925 07:21:30.824726 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:21:30.826365 14078 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:21:30.825933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.4.47.62:41417)
0925 07:21:30.825992 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:30.825993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:30.826099 (+ 106us) server_negotiation.cc:407] Connection header received
0925 07:21:30.826138 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:30.826139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:30.826143 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:30.826190 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:30.826289 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51304: BlockingRecv error: recv got EOF from 127.0.0.1:51304 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:21:30.826490 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14080
I20260925 07:21:30.831727 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14211
I20260925 07:21:30.836071 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14342
I20260925 07:21:30.840301 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 13989
2026-09-25T07:21:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (522 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:30 dist-test-slave-zw07 krb5kdc[14480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:30 dist-test-slave-zw07 krb5kdc[14480](info): set up 2 sockets
Sep 25 07:21:30 dist-test-slave-zw07 krb5kdc[14480](info): commencing operation
krb5kdc: starting...
W20260925 07:21:31.880402 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:31 dist-test-slave-zw07 krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:21:32.003816 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:43711
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:32.085357 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:32.085585 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:32.085608 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:32.086776 14493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:21:32.086818 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:32.086833 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:32.086845 14493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:21:32.086903 14493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:21:32.088352 14493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:43711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:32.088621 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:32.088857 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:32.093684 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:32.093690 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.093878 14493 server_base.cc:1034] running on GCE node
W20260925 07:21:32.093912 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.094133 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:32.094429 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:32.095592 14493 hybrid_clock.cc:648] HybridClock initialized: now 1790320892095588 us; error 38 us; skew 500 ppm
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:32.098100 14493 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:21:32.098537 14493 webserver.cc:460] Webserver started at http://127.4.47.62:45655/ using document root <none> and password file <none>
I20260925 07:21:32.098774 14493 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:32.098842 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:32.098989 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:32.100106 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "18db57b7a4ffb0259524c49dff665f5e"
server_key_iv: "d32129a16d240c1fab05cb0f3521723c"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.100486 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "18db57b7a4ffb0259524c49dff665f5e"
server_key_iv: "d32129a16d240c1fab05cb0f3521723c"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.101868 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:32.102797 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.102989 14493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:32.103112 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "18db57b7a4ffb0259524c49dff665f5e"
server_key_iv: "d32129a16d240c1fab05cb0f3521723c"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.103204 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:32.126056 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:32.128454 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:32.128628 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:32.132979 14493 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:43711
I20260925 07:21:32.133036 14560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:43711 every 8 connection(s)
I20260925 07:21:32.133436 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:21:32.134601 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:32.137327 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: Bootstrap starting.
I20260925 07:21:32.137933 14561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:32.138239 14561 log.cc:824] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:32.138674 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14493
I20260925 07:21:32.138805 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:21:32.138986 4284 external_mini_cluster.cc:1428] Setting key 32f17d9d8ed59a0fbf0eeeb7d54c7574
I20260925 07:21:32.139128 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: No bootstrap required, opened a new log
I20260925 07:21:32.141034 14561 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } }
I20260925 07:21:32.141211 14561 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:32.141256 14561 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e1e6e2d1d6450c9c7ddd5a4b2c692c, State: Initialized, Role: FOLLOWER
I20260925 07:21:32.141414 14561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } }
I20260925 07:21:32.141527 14561 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:21:32.141572 14561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:21:32.141630 14561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:32.142656 14561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } }
I20260925 07:21:32.142745 14561 leader_election.cc:302] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e1e6e2d1d6450c9c7ddd5a4b2c692c; no voters:
I20260925 07:21:32.142943 14561 leader_election.cc:288] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:21:32.142966 14566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:32.143100 14566 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [term 1 LEADER]: Becoming Leader. State: Replica: f1e1e6e2d1d6450c9c7ddd5a4b2c692c, State: Running, Role: LEADER
I20260925 07:21:32.143190 14566 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } }
I20260925 07:21:32.143230 14561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:21:32.143491 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } } }
I20260925 07:21:32.143548 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:32.143671 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e1e6e2d1d6450c9c7ddd5a4b2c692c. Latest consensus state: current_term: 1 leader_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e1e6e2d1d6450c9c7ddd5a4b2c692c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43711 } } }
I20260925 07:21:32.143728 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c [sys.catalog]: This master's current role is: LEADER
I20260925 07:21:32.144346 14570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:21:32.144927 14570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:21:32.146735 14570 catalog_manager.cc:1347] Generated new cluster ID: 2243a2c0321649978872eec609e1c6cf
I20260925 07:21:32.146812 14570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:32.151750 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.139600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.4.47.62:43711)
0925 07:21:32.139789 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.139799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.139833 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:21:32.141022 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.141031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.141090 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.141264 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:32.142018 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.143047 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.144047 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.144672 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.144921 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.147843 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:32.147847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:32.147847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:32.147859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:32.149867 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.150277 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.150280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.150281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.150425 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.150609 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.150610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.150610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.150924 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:32.151107 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.151383 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.151539 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260925 07:21:32.173096 14570 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:21:32.173691 14570 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:21:32.179653 14570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: Generated new TSK 0
I20260925 07:21:32.179872 14570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:21:32.214645 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:32.297500 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:32.297706 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:32.297729 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:32.298851 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:32.298887 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:32.298911 14587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:21:32.300393 14587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:32.300580 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:32.300750 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:32.305717 14592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:32.305785 14593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.305855 14587 server_base.cc:1034] running on GCE node
W20260925 07:21:32.305992 14595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.306205 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:32.306468 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:32.307629 14587 hybrid_clock.cc:648] HybridClock initialized: now 1790320892307642 us; error 55 us; skew 500 ppm
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:32.310220 14587 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:21:32.310706 14587 webserver.cc:460] Webserver started at http://127.4.47.1:36091/ using document root <none> and password file <none>
I20260925 07:21:32.310961 14587 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:32.311029 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:32.311214 14587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:32.312296 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "ca86b8ea0b5b41f88c5c20123aef6abf"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "d759eebd644aaf1946157d333315d45e"
server_key_iv: "d5b1fd9cb93eef3aa26ca7741c61f01a"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.312673 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "ca86b8ea0b5b41f88c5c20123aef6abf"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "d759eebd644aaf1946157d333315d45e"
server_key_iv: "d5b1fd9cb93eef3aa26ca7741c61f01a"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.314059 14587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:21:32.314841 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.315109 14587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:21:32.315217 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "ca86b8ea0b5b41f88c5c20123aef6abf"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "d759eebd644aaf1946157d333315d45e"
server_key_iv: "d5b1fd9cb93eef3aa26ca7741c61f01a"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.315305 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:32.331861 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:32.333972 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:32.334152 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:32.334408 14587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:32.334745 14587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:32.334801 14587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.334834 14587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:32.334864 14587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.341483 14587 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:34997
I20260925 07:21:32.341500 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:34997 every 8 connection(s)
I20260925 07:21:32.341879 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:32.349956 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14587
I20260925 07:21:32.350056 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:21:32.350292 4284 external_mini_cluster.cc:1428] Setting key fd73c4974e6085336c3f5719193ffe74
I20260925 07:21:32.352532 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.342678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:35549 (local address 127.4.47.62:43711)
0925 07:21:32.342741 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.342744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.342952 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:21:32.343657 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.343659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.343673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.343729 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:32.344792 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.345499 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.346556 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.346785 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.346875 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.349152 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:32.349153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:32.349153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:32.349160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:32.350848 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.351358 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.351359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.351360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.351424 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.351717 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.351718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.351719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.351935 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:32.352001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.352302 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.352398 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:21:32.352628 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.342659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:43711 (local address 127.4.47.1:35549)
0925 07:21:32.342825 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.342865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.343516 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.343803 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.343807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.344475 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:32.344692 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.344698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.345603 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.345604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.346444 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.346446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.346887 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.346888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.347016 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.347577 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:32.347586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:32.349054 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:32.350933 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.350941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.350944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.351282 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.351503 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.351504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.351504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.351616 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.352020 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:32.352025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:32.352156 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.352258 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.352390 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260925 07:21:32.353135 14716 heartbeater.cc:347] Connected to a master server at 127.4.47.62:43711
I20260925 07:21:32.353250 14716 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:32.353456 14716 heartbeater.cc:511] Master 127.4.47.62:43711 requested a full tablet report, sending...
I20260925 07:21:32.354085 14525 ts_manager.cc:194] Registered new tserver with Master: ca86b8ea0b5b41f88c5c20123aef6abf (127.4.47.1:34997)
I20260925 07:21:32.354843 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:35549
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:21:32.377667 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:32.460350 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:32.460565 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:32.460590 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:32.461804 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:32.461849 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:32.461876 14721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:21:32.463356 14721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:32.463616 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:32.463850 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:32.468885 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:32.469017 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.468971 14721 server_base.cc:1034] running on GCE node
W20260925 07:21:32.468914 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.469304 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:32.469539 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:32.470697 14721 hybrid_clock.cc:648] HybridClock initialized: now 1790320892470684 us; error 46 us; skew 500 ppm
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:32.473278 14721 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:21:32.473809 14721 webserver.cc:460] Webserver started at http://127.4.47.2:42397/ using document root <none> and password file <none>
I20260925 07:21:32.474052 14721 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:32.474097 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:32.474252 14721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:32.475616 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "8ee0c9a9522643609edc7c31e26f4823"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "035bff11c26f24c31d9a4178e5f4894a"
server_key_iv: "d0b9ba7f467e19f301f8931feb7d784d"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.475912 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "8ee0c9a9522643609edc7c31e26f4823"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "035bff11c26f24c31d9a4178e5f4894a"
server_key_iv: "d0b9ba7f467e19f301f8931feb7d784d"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.477411 14721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:21:32.478091 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.478247 14721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:21:32.478336 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "8ee0c9a9522643609edc7c31e26f4823"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "035bff11c26f24c31d9a4178e5f4894a"
server_key_iv: "d0b9ba7f467e19f301f8931feb7d784d"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.478397 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:32.505851 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:32.508141 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:32.508272 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:32.508473 14721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:32.508816 14721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:32.508850 14721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.508873 14721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:32.508886 14721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.514820 14721 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:32849
I20260925 07:21:32.514859 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:32849 every 8 connection(s)
I20260925 07:21:32.515259 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:32.523044 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14721
I20260925 07:21:32.523187 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:21:32.523358 4284 external_mini_cluster.cc:1428] Setting key 2971d53be8450ee937b06b52cfdea360
I20260925 07:21:32.524521 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.516004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56115 (local address 127.4.47.62:43711)
0925 07:21:32.516088 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.516090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.516242 (+ 152us) server_negotiation.cc:407] Connection header received
0925 07:21:32.516712 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.516713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.516723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.516773 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:32.517676 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.518178 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.519265 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.519456 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.519517 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.521737 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:32.521739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:32.521739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:32.521746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:32.522918 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.523447 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.523448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.523448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.523534 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.523847 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.523848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.523848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.523983 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:32.524053 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.524389 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.524458 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:21:32.524818 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.515954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:43711 (local address 127.4.47.2:56115)
0925 07:21:32.516120 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.516155 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.516623 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.516827 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.516830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.517392 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:32.517595 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.517599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.518238 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.518239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.519189 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.519192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.519565 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.519566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.519665 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.520225 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:32.520233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:32.521630 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:32.523008 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.523015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.523017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.523354 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.523610 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.523611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.523611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.523767 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.524073 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:32.524080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:32.524246 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.524367 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.524537 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260925 07:21:32.525309 14850 heartbeater.cc:347] Connected to a master server at 127.4.47.62:43711
I20260925 07:21:32.525405 14850 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:32.525601 14850 heartbeater.cc:511] Master 127.4.47.62:43711 requested a full tablet report, sending...
I20260925 07:21:32.525997 14525 ts_manager.cc:194] Registered new tserver with Master: 8ee0c9a9522643609edc7c31e26f4823 (127.4.47.2:32849)
I20260925 07:21:32.526384 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:56115
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:21:32.549880 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:21:32.639860 14855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:21:32.640090 14855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:21:32.640116 14855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:21:32.641314 14855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:21:32.641351 14855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:21:32.641379 14855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:21:32.642897 14855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:21:32.643216 14855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:21:32.643524 14855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:21:32.649334 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:32.649353 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:21:32.649346 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:21:32.649618 14855 server_base.cc:1034] running on GCE node
I20260925 07:21:32.649869 14855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:21:32.650130 14855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:21:32.651278 14855 hybrid_clock.cc:648] HybridClock initialized: now 1790320892651250 us; error 50 us; skew 500 ppm
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:21:32.653892 14855 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:21:32.654340 14855 webserver.cc:460] Webserver started at http://127.4.47.3:40337/ using document root <none> and password file <none>
I20260925 07:21:32.654572 14855 fs_manager.cc:359] Metadata directory not provided
I20260925 07:21:32.654640 14855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:21:32.654794 14855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:21:32.655962 14855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "5c7327cffb534351b6f5cc94ec3b7963"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "47a4cc000612ffb426d6168d958d2027"
server_key_iv: "584930b2a04c0fce88007f2bdf05112f"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.656333 14855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7327cffb534351b6f5cc94ec3b7963"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "47a4cc000612ffb426d6168d958d2027"
server_key_iv: "584930b2a04c0fce88007f2bdf05112f"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.657748 14855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:21:32.658504 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.658691 14855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:21:32.658798 14855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "5c7327cffb534351b6f5cc94ec3b7963"
format_stamp: "Formatted at 2026-09-25 07:21:32 on dist-test-slave-zw07"
server_key: "47a4cc000612ffb426d6168d958d2027"
server_key_iv: "584930b2a04c0fce88007f2bdf05112f"
server_key_version: "encryptionkey@0"
I20260925 07:21:32.658874 14855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:21:32.677273 14855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:21:32.679432 14855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:21:32.679629 14855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:21:32.679939 14855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:21:32.680320 14855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:21:32.680357 14855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.680380 14855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:21:32.680428 14855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:21:32.686570 14855 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:41707
I20260925 07:21:32.686612 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:41707 every 8 connection(s)
I20260925 07:21:32.686954 14855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:21:32 dist-test-slave-zw07 krb5kdc[14480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320892, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:21:32.695451 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 14855
I20260925 07:21:32.695618 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:21:32.695855 4284 external_mini_cluster.cc:1428] Setting key 6d8ee62a2c38d59e0cfc3ca7bfa70a0d
I20260925 07:21:32.696831 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.687771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:56735 (local address 127.4.47.62:43711)
0925 07:21:32.687843 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.687863 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.688055 (+ 192us) server_negotiation.cc:407] Connection header received
0925 07:21:32.688631 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.688632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.688645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.688703 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:32.689726 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.690230 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.691340 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.691535 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.691638 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.693831 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:32.693833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:32.693833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:32.693840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:32.695130 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.695703 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.695705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.695705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.695773 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.696105 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.696107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.696107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.696283 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:32.696356 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.696652 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.696737 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:21:32.697031 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.687766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:43711 (local address 127.4.47.3:56735)
0925 07:21:32.687923 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.687969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.688506 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.688775 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.688779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.689386 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:21:32.689588 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.689594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.690331 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.690332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.691238 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.691241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.691646 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.691647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.691788 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.692310 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:21:32.692318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:21:32.693740 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:21:32.695216 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.695222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.695224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.695595 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.695853 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:21:32.695853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:21:32.695854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:21:32.696011 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:21:32.696365 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:21:32.696372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:21:32.696514 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.696629 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.696798 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260925 07:21:32.697576 14984 heartbeater.cc:347] Connected to a master server at 127.4.47.62:43711
I20260925 07:21:32.697674 14984 heartbeater.cc:464] Registering TS with master...
I20260925 07:21:32.697842 14984 heartbeater.cc:511] Master 127.4.47.62:43711 requested a full tablet report, sending...
I20260925 07:21:32.698230 14525 ts_manager.cc:194] Registered new tserver with Master: 5c7327cffb534351b6f5cc94ec3b7963 (127.4.47.3:41707)
I20260925 07:21:32.698596 14525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:56735
I20260925 07:21:32.707921 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:21:32.715695 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.709366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.4.47.62:43711)
0925 07:21:32.709460 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.709462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.709564 (+ 102us) server_negotiation.cc:407] Connection header received
0925 07:21:32.709640 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.709641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.709652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.709699 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:21:32.710310 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.710811 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.711642 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.711848 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.711931 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.713182 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:21:32.713184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:21:32.713184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:21:32.713192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:21:32.714377 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.714745 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.714746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.714747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.714804 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:21:32.715023 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:21:32.715024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:21:32.715024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:21:32.715222 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:21:32.715345 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.715449 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.715586 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 07:21:32.717540 14525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:21:32.719282 14525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:21:32.726141 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.721681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.4.47.3:41707)
0925 07:21:32.721864 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.721869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.722226 (+ 357us) server_negotiation.cc:407] Connection header received
0925 07:21:32.722276 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.722279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.722297 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.722623 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:32.722702 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.723543 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.724979 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.725661 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.725755 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.725821 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.726003 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.726049 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 07:21:32.726136 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.721591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:41707 (local address 127.0.0.1:44840)
0925 07:21:32.721994 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.722039 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.722116 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.722456 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.722459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.722461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:32.722544 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.722546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.723644 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.723645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.724885 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.724888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.725753 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.725754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.725844 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.725860 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.725983 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.726068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":67,"threads_started":1}
I20260925 07:21:32.726524 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.721595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34997 (local address 127.0.0.1:35738)
0925 07:21:32.722239 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.722250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.722327 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.722598 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.722599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.722602 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:32.722657 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.722658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.724025 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.724026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.725060 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.725068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.726161 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.726162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.726334 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.726340 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.726398 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.726444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":80,"threads_started":1}
I20260925 07:21:32.726552 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.722003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.4.47.1:34997)
0925 07:21:32.722192 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.722197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.722337 (+ 140us) server_negotiation.cc:407] Connection header received
0925 07:21:32.722405 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.722407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.722424 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.722512 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:32.722722 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.723893 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.725179 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.725975 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.726050 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.726123 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.726430 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.726474 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260925 07:21:32.726840 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.721833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:32849 (local address 127.0.0.1:56270)
0925 07:21:32.722399 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:32.722407 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:32.722448 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:32.722730 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:32.722730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:32.722733 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:32.722775 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.722776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.724063 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.724064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.725096 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:32.725098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.726610 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.726611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:32.726676 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.726682 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:21:32.726725 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 07:21:32.726766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260925 07:21:32.727150 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:32.721945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.4.47.2:32849)
0925 07:21:32.722108 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:32.722112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:32.722507 (+ 395us) server_negotiation.cc:407] Connection header received
0925 07:21:32.722551 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:32.722553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:32.722569 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:32.722634 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:32.722856 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.723949 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.725330 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:32.726097 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:32.726162 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:32.726963 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:32.726990 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 07:21:32.727029 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260925 07:21:32.727070 14918 tablet_service.cc:1467] Processing CreateTablet for tablet d962758198b245fc90e07dccb8107889 (DEFAULT_TABLE table=test-table [id=68cfdf25fd934e3583b418da29143cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:32.727267 14650 tablet_service.cc:1467] Processing CreateTablet for tablet d962758198b245fc90e07dccb8107889 (DEFAULT_TABLE table=test-table [id=68cfdf25fd934e3583b418da29143cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:32.727469 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d962758198b245fc90e07dccb8107889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:32.727530 14650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d962758198b245fc90e07dccb8107889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:32.727712 14784 tablet_service.cc:1467] Processing CreateTablet for tablet d962758198b245fc90e07dccb8107889 (DEFAULT_TABLE table=test-table [id=68cfdf25fd934e3583b418da29143cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:21:32.727960 14784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d962758198b245fc90e07dccb8107889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:21:32.730346 15001 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Bootstrap starting.
I20260925 07:21:32.730827 15002 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Bootstrap starting.
I20260925 07:21:32.731318 15001 tablet_bootstrap.cc:654] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:32.731518 15002 tablet_bootstrap.cc:654] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:32.731937 15001 log.cc:824] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:32.732174 15002 log.cc:824] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:32.732342 15003 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Bootstrap starting.
I20260925 07:21:32.732939 15002 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: No bootstrap required, opened a new log
I20260925 07:21:32.732944 15001 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: No bootstrap required, opened a new log
I20260925 07:21:32.732980 15003 tablet_bootstrap.cc:654] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Neither blocks nor log segments found. Creating new log.
I20260925 07:21:32.733006 15001 ts_tablet_manager.cc:1397] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:21:32.733024 15002 ts_tablet_manager.cc:1397] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:21:32.733510 15003 log.cc:824] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:21:32.734221 15003 tablet_bootstrap.cc:492] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: No bootstrap required, opened a new log
I20260925 07:21:32.734303 15003 ts_tablet_manager.cc:1397] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:21:32.734767 15002 raft_consensus.cc:348] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.734927 15002 raft_consensus.cc:374] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:32.734968 15002 raft_consensus.cc:729] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7327cffb534351b6f5cc94ec3b7963, State: Initialized, Role: FOLLOWER
I20260925 07:21:32.735138 15002 consensus_queue.cc:260] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.735255 15001 raft_consensus.cc:348] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.735421 15001 raft_consensus.cc:374] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:32.735438 15002 ts_tablet_manager.cc:1428] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:21:32.735467 15001 raft_consensus.cc:729] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee0c9a9522643609edc7c31e26f4823, State: Initialized, Role: FOLLOWER
I20260925 07:21:32.735576 14984 heartbeater.cc:503] Master 127.4.47.62:43711 was elected leader, sending a full tablet report...
I20260925 07:21:32.735602 15001 consensus_queue.cc:260] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.735872 15001 ts_tablet_manager.cc:1428] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:21:32.736272 15003 raft_consensus.cc:348] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.736429 15003 raft_consensus.cc:374] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:21:32.736477 15003 raft_consensus.cc:729] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca86b8ea0b5b41f88c5c20123aef6abf, State: Initialized, Role: FOLLOWER
I20260925 07:21:32.736591 15003 consensus_queue.cc:260] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:32.736877 15003 ts_tablet_manager.cc:1428] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:21:32.736891 14716 heartbeater.cc:503] Master 127.4.47.62:43711 was elected leader, sending a full tablet report...
I20260925 07:21:32.736994 14850 heartbeater.cc:503] Master 127.4.47.62:43711 was elected leader, sending a full tablet report...
W20260925 07:21:32.765866 14851 tablet.cc:2369] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:21:32.842671 14717 tablet.cc:2369] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:21:32.937615 14985 tablet.cc:2369] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:21:33.000272 15007 raft_consensus.cc:482] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:21:33.000470 15007 raft_consensus.cc:504] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:33.000897 15007 leader_election.cc:288] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca86b8ea0b5b41f88c5c20123aef6abf (127.4.47.1:34997), 8ee0c9a9522643609edc7c31e26f4823 (127.4.47.2:32849)
I20260925 07:21:33.005121 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.001188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:57623 (local address 127.4.47.1:34997)
0925 07:21:33.001237 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:33.001239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:33.001474 (+ 235us) server_negotiation.cc:407] Connection header received
0925 07:21:33.001542 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:33.001544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:33.001556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:33.001616 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:33.001768 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.002513 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004158 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.004736 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:33.004830 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:33.005007 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 07:21:33.005047 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:21:33.005121 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.001202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:49229 (local address 127.4.47.2:32849)
0925 07:21:33.001244 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:33.001246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:33.001283 (+ 37us) server_negotiation.cc:407] Connection header received
0925 07:21:33.001392 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:33.001393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:33.001405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:33.001499 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:21:33.001746 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.002519 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004158 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.004736 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:33.004830 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:33.005007 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 07:21:33.005047 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:21:33.005129 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.001114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:32849 (local address 127.4.47.3:49229)
0925 07:21:33.001210 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:33.001224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:33.001304 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:33.001528 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:33.001529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:33.001530 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:33.001638 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:33.001640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.002630 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.002631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:33.004030 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:33.004035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.004835 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:33.004920 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:33.004922 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:21:33.004981 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:21:33.005030 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":89}
I20260925 07:21:33.005137 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.001122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34997 (local address 127.4.47.3:57623)
0925 07:21:33.001390 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:33.001399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:21:33.001462 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:21:33.001634 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:21:33.001635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:21:33.001636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:21:33.001694 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:33.001695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.002630 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.002631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:33.004030 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:21:33.004035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.004835 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.004837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:21:33.004917 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:33.004921 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:21:33.004981 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 07:21:33.005030 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":115,"thread_start_us":139,"threads_started":1}
I20260925 07:21:33.005556 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d962758198b245fc90e07dccb8107889" candidate_uuid: "5c7327cffb534351b6f5cc94ec3b7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" is_pre_election: true
I20260925 07:21:33.005564 14804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d962758198b245fc90e07dccb8107889" candidate_uuid: "5c7327cffb534351b6f5cc94ec3b7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee0c9a9522643609edc7c31e26f4823" is_pre_election: true
I20260925 07:21:33.005744 14804 raft_consensus.cc:2454] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7327cffb534351b6f5cc94ec3b7963 in term 0.
I20260925 07:21:33.005744 14670 raft_consensus.cc:2454] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7327cffb534351b6f5cc94ec3b7963 in term 0.
I20260925 07:21:33.005999 14873 leader_election.cc:302] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7327cffb534351b6f5cc94ec3b7963, ca86b8ea0b5b41f88c5c20123aef6abf; no voters:
I20260925 07:21:33.006170 15007 raft_consensus.cc:2781] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:21:33.006224 15007 raft_consensus.cc:482] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:21:33.006248 15007 raft_consensus.cc:3037] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:33.007036 15007 raft_consensus.cc:504] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:33.007243 15007 leader_election.cc:288] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [CANDIDATE]: Term 1 election: Requested vote from peers ca86b8ea0b5b41f88c5c20123aef6abf (127.4.47.1:34997), 8ee0c9a9522643609edc7c31e26f4823 (127.4.47.2:32849)
I20260925 07:21:33.007444 14804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d962758198b245fc90e07dccb8107889" candidate_uuid: "5c7327cffb534351b6f5cc94ec3b7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee0c9a9522643609edc7c31e26f4823"
I20260925 07:21:33.007442 14670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d962758198b245fc90e07dccb8107889" candidate_uuid: "5c7327cffb534351b6f5cc94ec3b7963" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf"
I20260925 07:21:33.007527 14804 raft_consensus.cc:3037] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:33.007527 14670 raft_consensus.cc:3037] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:21:33.008433 14804 raft_consensus.cc:2454] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7327cffb534351b6f5cc94ec3b7963 in term 1.
I20260925 07:21:33.008433 14670 raft_consensus.cc:2454] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7327cffb534351b6f5cc94ec3b7963 in term 1.
I20260925 07:21:33.008628 14874 leader_election.cc:302] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7327cffb534351b6f5cc94ec3b7963, 8ee0c9a9522643609edc7c31e26f4823; no voters:
I20260925 07:21:33.008713 15007 raft_consensus.cc:2781] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:21:33.008874 15007 raft_consensus.cc:686] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 1 LEADER]: Becoming Leader. State: Replica: 5c7327cffb534351b6f5cc94ec3b7963, State: Running, Role: LEADER
I20260925 07:21:33.009028 15007 consensus_queue.cc:237] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } }
I20260925 07:21:33.009903 14523 catalog_manager.cc:5495] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c7327cffb534351b6f5cc94ec3b7963 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "5c7327cffb534351b6f5cc94ec3b7963" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7327cffb534351b6f5cc94ec3b7963" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 } health_report { overall_health: UNKNOWN } } }
I20260925 07:21:33.083503 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.080925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.4.47.3:41707)
0925 07:21:33.080998 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:21:33.081000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:21:33.081010 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:21:33.081051 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:21:33.081052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:21:33.081074 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:21:33.081169 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:21:33.081342 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.081851 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.082627 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:21:33.082850 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:21:33.082927 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:21:33.083126 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:21:33.083245 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:21:33.083350 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 07:21:33.083369 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 07:21:33.083390 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:21:33.085403 14804 raft_consensus.cc:1261] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 1 FOLLOWER]: Refusing update from remote peer 5c7327cffb534351b6f5cc94ec3b7963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:33.085386 14670 raft_consensus.cc:1261] T d962758198b245fc90e07dccb8107889 P ca86b8ea0b5b41f88c5c20123aef6abf [term 1 FOLLOWER]: Refusing update from remote peer 5c7327cffb534351b6f5cc94ec3b7963: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:21:33.085765 15007 consensus_queue.cc:1035] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca86b8ea0b5b41f88c5c20123aef6abf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:33.085929 15011 consensus_queue.cc:1035] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee0c9a9522643609edc7c31e26f4823" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:21:33.088392 15014 mvcc.cc:205] Tried to move back new op lower bound from 7333154378074636288 to 7333154377765543936. Current Snapshot: MvccSnapshot[applied={T|T < 7333154378074636288}]
I20260925 07:21:33.088930 15015 mvcc.cc:205] Tried to move back new op lower bound from 7333154378074636288 to 7333154377765543936. Current Snapshot: MvccSnapshot[applied={T|T < 7333154378074636288}]
I20260925 07:21:33.090926 14524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:33.091049 14524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:21:33.091859 14524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e1e6e2d1d6450c9c7ddd5a4b2c692c: Sending DeleteTablet for 3 replicas of tablet d962758198b245fc90e07dccb8107889
I20260925 07:21:33.092238 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet d962758198b245fc90e07dccb8107889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260925 07:21:33.092238 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet d962758198b245fc90e07dccb8107889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:56270
I20260925 07:21:33.092257 14650 tablet_service.cc:1514] Processing DeleteTablet for tablet d962758198b245fc90e07dccb8107889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:21:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:35738
I20260925 07:21:33.092479 15024 tablet_replica.cc:330] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: stopping tablet replica
I20260925 07:21:33.092576 15024 raft_consensus.cc:2229] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:21:33.092661 15025 tablet_replica.cc:330] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: stopping tablet replica
I20260925 07:21:33.092691 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14587
I20260925 07:21:33.092698 15024 raft_consensus.cc:2258] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:33.092795 15025 raft_consensus.cc:2229] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:21:33.092911 15025 raft_consensus.cc:2258] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:21:33.093713 15025 ts_tablet_manager.cc:1905] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:33.094303 15024 ts_tablet_manager.cc:1905] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:21:33.095774 15025 ts_tablet_manager.cc:1918] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:33.095870 15025 log.cc:1196] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/d962758198b245fc90e07dccb8107889
I20260925 07:21:33.095973 15024 ts_tablet_manager.cc:1918] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:21:33.096047 15024 log.cc:1196] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/d962758198b245fc90e07dccb8107889
I20260925 07:21:33.096196 15025 ts_tablet_manager.cc:1939] T d962758198b245fc90e07dccb8107889 P 8ee0c9a9522643609edc7c31e26f4823: Deleting consensus metadata
I20260925 07:21:33.096340 15024 ts_tablet_manager.cc:1939] T d962758198b245fc90e07dccb8107889 P 5c7327cffb534351b6f5cc94ec3b7963: Deleting consensus metadata
I20260925 07:21:33.096745 14512 catalog_manager.cc:4841] TS 5c7327cffb534351b6f5cc94ec3b7963 (127.4.47.3:41707): tablet d962758198b245fc90e07dccb8107889 (table test-table [id=68cfdf25fd934e3583b418da29143cf1]) successfully deleted
I20260925 07:21:33.096848 14512 catalog_manager.cc:4841] TS 8ee0c9a9522643609edc7c31e26f4823 (127.4.47.2:32849): tablet d962758198b245fc90e07dccb8107889 (table test-table [id=68cfdf25fd934e3583b418da29143cf1]) successfully deleted
W20260925 07:21:33.102314 14511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:34997 (error 108)
I20260925 07:21:33.102491 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14721
I20260925 07:21:33.103374 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:21:33.103246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34997 (local address 127.0.0.1:35752)
0925 07:21:33.103294 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:21:33.103332 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:21:33.103508 14511 catalog_manager.cc:4584] TS ca86b8ea0b5b41f88c5c20123aef6abf (127.4.47.1:34997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d962758198b245fc90e07dccb8107889: Network error: Client connection negotiation failed: client connection to 127.4.47.1:34997: connect: Connection refused (error 111)
I20260925 07:21:33.108191 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14855
I20260925 07:21:33.113204 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 14493
2026-09-25T07:21:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2273 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:21:33 dist-test-slave-zw07 krb5kdc[15030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:21:33 dist-test-slave-zw07 krb5kdc[15030](info): set up 2 sockets
Sep 25 07:21:33 dist-test-slave-zw07 krb5kdc[15030](info): commencing operation
krb5kdc: starting...
W20260925 07:21:34.164162 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:21:34 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:21:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangeradmin_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangeradmin_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangerlookup_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangerlookup_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
I20260925 07:21:34.330065 4284 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSRTLta/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/postgres -l logfile start
2026-09-25 07:21:36.888 UTC [15060] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 07:21:36.888 UTC [15060] LOG: listening on IPv4 address "127.4.47.20", port 33093
2026-09-25 07:21:36.892 UTC [15060] LOG: listening on Unix socket "/tmp/.s.PGSQL.33093"
2026-09-25 07:21:36.927 UTC [15063] LOG: database system was shut down at 2026-09-25 07:21:35 UTC
2026-09-25 07:21:36.931 UTC [15060] LOG: database system is ready to accept connections
I20260925 07:21:37.903889 4284 mini_postgres.cc:96] Postgres bound to 33093
2026-09-25 07:21:37.910 UTC [15071] FATAL: database "slave" does not exist
127.4.47.20:33093 - accepting connections
I20260925 07:21:37.911139 4284 mini_ranger.cc:161] Starting Ranger...
I20260925 07:21:37.925899 4284 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 07:21:38.442142 4284 mini_ranger.cc:87] Created ranger Postgres database
I20260925 07:21:38.442232 4284 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin
2026-09-25 07:21:38,730 [I] DB FLAVOR :POSTGRES
2026-09-25 07:21:38,730 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:21:38,730 [I] Checking connection..
2026-09-25 07:21:38,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 07:21:38,996 [I] Checking connection passed.
2026-09-25 07:21:38,996 [I] --------- Verifying version history table ---------
2026-09-25 07:21:38,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:21:39,265 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:21:39,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:21:39,506 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:21:39,506 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 07:21:39,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 07:21:39.758 UTC [15135] WARNING: there is no transaction in progress
2026-09-25 07:21:39,764 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 07:21:39,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:21:40,014 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 07:21:40,015 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:21:40,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 07:21:40,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 07:21:40,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zw07.c.gcp-upstream.internal','N') ;"
2026-09-25 07:21:40,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:21:41,018 [I] Table x_portal_user does not exist in database ranger
2026-09-25 07:21:41,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:21:41,256 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 07:21:41,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:21:41,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:21:41,762 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 07:21:41,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 07:21:42.766 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.784 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.802 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.820 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.839 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.861 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.930 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.938 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.953 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.964 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.975 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:42.997 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:43.012 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:43.020 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:43.028 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:43.036 UTC [15243] WARNING: there is no transaction in progress
2026-09-25 07:21:43,732 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 07:21:43,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 07:21:43,958 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 07:21:43,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:21:44,210 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 07:21:44,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:21:44,447 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 07:21:44,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:21:44,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:21:44,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zw07.c.gcp-upstream.internal';"
2026-09-25 07:21:45,181 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 07:21:45,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 07:21:45,443 [I] DB_PATCHES have already been applied
I20260925 07:21:45.449184 4284 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSRTLta/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 07:21:45.449272 4284 mini_ranger.cc:193] Using host: 127.4.47.20
I20260925 07:21:45.452201 4284 mini_ranger.cc:227] Ranger admin URL: http://127.4.47.20:56255
Sep 25, 2026 7:21:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:21:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:21:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:21:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:21:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:21:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:21:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 07:21:46.561087 4284 mini_ranger.cc:160] Time spent starting Ranger: real 8.650s user 0.000s sys 0.009s
Sep 25, 2026 7:22:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:22:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 7:22:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:22:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:22:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 07:22:08.921543 4284 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangerkms_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/rangerkms_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.47.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.47.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 07:22:08 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 07:22:09.007998 4284 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 07:22:09.018296 4284 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 07:22:09.942822 4284 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 07:22:09.942912 4284 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms
I20260925 07:22:09.942934 4284 mini_ranger_kms.cc:227] Using postgres at 127.4.47.20:33093
2026-09-25 07:22:10,119 [I] DB FLAVOR :POSTGRES
2026-09-25 07:22:10,119 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:22:10,119 [I] Checking connection
2026-09-25 07:22:10,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 07:22:10,373 [I] connection success
2026-09-25 07:22:10,374 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 07:22:10,374 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 07:22:10,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 07:22:10,604 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 07:22:10,605 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:22:10,605 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 07:22:10,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:33093/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 07:22:10,872 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 07:22:11.320525 4284 mini_ranger_kms.cc:310] Created kms service
I20260925 07:22:11.490437 4284 mini_ranger_kms.cc:326] Created kudu user
I20260925 07:22:11.535518 4284 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 07:22:11 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:11.961971 4284 mini_ranger_kms.cc:384] Added ranger policy
I20260925 07:22:11.962087 4284 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 07:22:11.962109 4284 mini_ranger_kms.cc:240] Using host: 127.4.47.20
I20260925 07:22:11.965430 4284 mini_ranger_kms.cc:276] Ranger KMS PID: 15423
I20260925 07:22:11.965569 4284 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.47.20:47961
Sep 25, 2026 7:22:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:22:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:22:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:22:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:22:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:22:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 07:22:14 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790320934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:14.232117 4284 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.224s user 0.000s sys 0.159s
I20260925 07:22:14.232245 4284 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 07:22:14.232340 4284 mini_ranger_kms.cc:401] 127.4.47.20:47961/kms/v1/keys
Sep 25 07:22:14 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
Sep 25 07:22:14 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
Sep 25 07:22:17 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:22:17.210628 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41173
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:41173
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:47961/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:22:17.292384 15474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:22:17.292610 15474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:22:17.292632 15474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:22:17.293692 15474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:22:17.293728 15474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:22:17.293742 15474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:22:17.293756 15474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:22:17.293767 15474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:22:17.295352 15474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.47.20:47961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:41173
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:41173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:22:17.295543 15474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:22:17.295703 15474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:22:17.300724 15474 server_base.cc:1034] running on GCE node
W20260925 07:22:17.300757 15480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:17.300836 15482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:17.300758 15479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:22:17.301149 15474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:22:17.301433 15474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:22:17.302587 15474 hybrid_clock.cc:648] HybridClock initialized: now 1790320937302575 us; error 43 us; skew 500 ppm
Sep 25 07:22:17 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320937, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:17.312441 15474 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:22:17.312912 15474 webserver.cc:460] Webserver started at http://127.4.47.62:39611/ using document root <none> and password file <none>
I20260925 07:22:17.313092 15474 fs_manager.cc:359] Metadata directory not provided
I20260925 07:22:17.313128 15474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:22:17.313218 15474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:22:17 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320937, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:17.360618 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "ec5f09d0d1f74b879f67238bcfeb8d67"
format_stamp: "Formatted at 2026-09-25 07:22:17 on dist-test-slave-zw07"
server_key: "1a62e05ef740d207690b9663cc012da7"
server_key_iv: "f470fc64cc537399c9342f4341d497d0"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:17.361034 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "ec5f09d0d1f74b879f67238bcfeb8d67"
format_stamp: "Formatted at 2026-09-25 07:22:17 on dist-test-slave-zw07"
server_key: "1a62e05ef740d207690b9663cc012da7"
server_key_iv: "f470fc64cc537399c9342f4341d497d0"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:17.362504 15474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:22:17.374043 15489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:17.374274 15474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:22:17.374395 15474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "ec5f09d0d1f74b879f67238bcfeb8d67"
format_stamp: "Formatted at 2026-09-25 07:22:17 on dist-test-slave-zw07"
server_key: "1a62e05ef740d207690b9663cc012da7"
server_key_iv: "f470fc64cc537399c9342f4341d497d0"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:17.374503 15474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:22:17.387929 15474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:22:17.389986 15474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:22:17.390142 15474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:22:17.395236 15474 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:41173
I20260925 07:22:17.395303 15541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:41173 every 8 connection(s)
I20260925 07:22:17.395675 15474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:22:17.396340 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 15474
I20260925 07:22:17.396368 15542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:22:17.396451 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:22:17.396643 4284 external_mini_cluster.cc:1428] Setting key 3048ca74dd6af82d4321bc49e62b078d
I20260925 07:22:17.398900 15542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: Bootstrap starting.
I20260925 07:22:17.399578 15542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: Neither blocks nor log segments found. Creating new log.
I20260925 07:22:17.399901 15542 log.cc:824] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: Log is configured to *not* fsync() on all Append() calls
I20260925 07:22:17.400676 15542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: No bootstrap required, opened a new log
I20260925 07:22:17.402824 15542 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } }
I20260925 07:22:17.403002 15542 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:22:17.403045 15542 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5f09d0d1f74b879f67238bcfeb8d67, State: Initialized, Role: FOLLOWER
I20260925 07:22:17.403247 15542 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } }
I20260925 07:22:17.403347 15542 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:22:17.403402 15542 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:22:17.403478 15542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:22:17.404067 15542 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } }
I20260925 07:22:17.404187 15542 leader_election.cc:302] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec5f09d0d1f74b879f67238bcfeb8d67; no voters:
I20260925 07:22:17.404388 15542 leader_election.cc:288] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:22:17.404414 15547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:22:17.404695 15542 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:22:17.405586 15547 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [term 1 LEADER]: Becoming Leader. State: Replica: ec5f09d0d1f74b879f67238bcfeb8d67, State: Running, Role: LEADER
I20260925 07:22:17.405753 15547 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } }
Sep 25 07:22:17 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:22:17.406705 15547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec5f09d0d1f74b879f67238bcfeb8d67. Latest consensus state: current_term: 1 leader_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } } }
I20260925 07:22:17.406821 15547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [sys.catalog]: This master's current role is: LEADER
I20260925 07:22:17.407326 15548 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5f09d0d1f74b879f67238bcfeb8d67" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 41173 } } }
I20260925 07:22:17.407655 15561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:22:17.407948 15542 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 07:22:17.408232 15561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:22:17.407727 15548 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67 [sys.catalog]: This master's current role is: LEADER
I20260925 07:22:17.410171 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:17.397175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.4.47.62:41173)
0925 07:22:17.398857 (+ 1682us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:17.398868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:17.398904 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:22:17.399499 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:17.399509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:17.399574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:17.399735 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:22:17.400527 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:17.401955 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:17.403127 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:17.403775 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:17.403889 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:17.406520 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:22:17.406524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:22:17.406525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:22:17.406536 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:22:17.408514 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:17.408934 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:17.408937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:17.408938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:17.409040 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:17.409255 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:17.409256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:17.409256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:17.409548 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:22:17.409675 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:17.409829 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 07:22:17.409947 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622,"thread_start_us":71,"threads_started":1}
I20260925 07:22:17.411203 15561 catalog_manager.cc:1347] Generated new cluster ID: 2f1540df4ce24f5d9c080e0ea56f0e7a
I20260925 07:22:17.411273 15561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:22:17.424639 15561 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:22:17.425642 15561 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:22:17.430087 15561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: Generated new TSK 0
I20260925 07:22:17.430450 15561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:22:18.095816 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:47961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:22:18.190475 15587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:22:18.190668 15587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:22:18.190689 15587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:22:18.191910 15587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:22:18.191952 15587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:22:18.191978 15587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:22:18.193432 15587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.47.20:47961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:22:18.193648 15587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:22:18.193885 15587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:22:18.199002 15595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.199237 15593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.199002 15592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:22:18.199523 15587 server_base.cc:1034] running on GCE node
I20260925 07:22:18.199704 15587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:22:18.199950 15587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:22:18.201167 15587 hybrid_clock.cc:648] HybridClock initialized: now 1790320938201154 us; error 29 us; skew 500 ppm
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:18.203831 15587 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:22:18.204326 15587 webserver.cc:460] Webserver started at http://127.4.47.1:38155/ using document root <none> and password file <none>
I20260925 07:22:18.204567 15587 fs_manager.cc:359] Metadata directory not provided
I20260925 07:22:18.204632 15587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:22:18.204962 15587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:18.220319 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "4bfdcf08988a4af181cff05e36afa9a4"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "e98b146a5ce458544bbd7e0efb867c7b"
server_key_iv: "b3d354b072c3ede6c9562ac1504fbc37"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.220860 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "4bfdcf08988a4af181cff05e36afa9a4"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "e98b146a5ce458544bbd7e0efb867c7b"
server_key_iv: "b3d354b072c3ede6c9562ac1504fbc37"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.222729 15587 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 07:22:18.238263 15602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.238667 15587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 07:22:18.238853 15587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "4bfdcf08988a4af181cff05e36afa9a4"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "e98b146a5ce458544bbd7e0efb867c7b"
server_key_iv: "b3d354b072c3ede6c9562ac1504fbc37"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.238986 15587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:22:18.248418 15587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:22:18.250388 15587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:22:18.250535 15587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:22:18.250746 15587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:22:18.251047 15587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:22:18.251132 15587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.251200 15587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:22:18.251245 15587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.256345 15587 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:45331
I20260925 07:22:18.256779 15587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:22:18.262110 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 15587
I20260925 07:22:18.262231 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:22:18.262419 4284 external_mini_cluster.cc:1428] Setting key c3a13e4076ce727e61975424d1ac5651
I20260925 07:22:18.265300 15715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:45331 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:22:18.278225 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.267107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:48075 (local address 127.4.47.62:41173)
0925 07:22:18.268422 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.268425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.268434 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:22:18.268461 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.268462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.268474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.268541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:22:18.269356 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.270191 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.271744 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.271966 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.272054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.274989 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:22:18.274991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:22:18.274991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:22:18.274998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:22:18.276621 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.277071 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.277073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.277073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.277135 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.277415 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.277416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.277416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.277549 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:22:18.277606 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.278032 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.278118 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":100,"threads_started":1}
I20260925 07:22:18.278590 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.267091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41173 (local address 127.4.47.1:48075)
0925 07:22:18.267407 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.267455 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.268201 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.268680 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.268685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.268943 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:22:18.269231 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.269238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.270298 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.270299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.271655 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.271657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.272063 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.272064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.272212 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.272766 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:22:18.272774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:22:18.274884 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:22:18.276721 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.276728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.276730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.276950 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.277209 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.277209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.277210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.277339 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.277631 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:22:18.277637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:22:18.277892 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.278000 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.278148 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260925 07:22:18.279110 15716 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41173
I20260925 07:22:18.279212 15716 heartbeater.cc:464] Registering TS with master...
I20260925 07:22:18.279405 15716 heartbeater.cc:511] Master 127.4.47.62:41173 requested a full tablet report, sending...
I20260925 07:22:18.279932 15506 ts_manager.cc:194] Registered new tserver with Master: 4bfdcf08988a4af181cff05e36afa9a4 (127.4.47.1:45331)
I20260925 07:22:18.280525 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:48075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:22:18.302302 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:47961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:22:18.419836 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:22:18.420114 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:22:18.420149 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:22:18.422185 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:22:18.422254 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:22:18.422295 15724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:22:18.424590 15724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.47.20:47961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:22:18.424839 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:22:18.425181 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:22:18.431560 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.431567 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.431821 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:22:18.432093 15724 server_base.cc:1034] running on GCE node
I20260925 07:22:18.432296 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:22:18.432581 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:22:18.433755 15724 hybrid_clock.cc:648] HybridClock initialized: now 1790320938433734 us; error 34 us; skew 500 ppm
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:18.437036 15724 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:22:18.437646 15724 webserver.cc:460] Webserver started at http://127.4.47.2:40133/ using document root <none> and password file <none>
I20260925 07:22:18.437911 15724 fs_manager.cc:359] Metadata directory not provided
I20260925 07:22:18.437974 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:22:18.438150 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:18.454907 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "a29f0487e06046eea676ef10c453bf02"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "dd5b595e3cd9718a14159bc724cc60e0"
server_key_iv: "bc30109831e6c98826bc64519bb8d503"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.455502 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "a29f0487e06046eea676ef10c453bf02"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "dd5b595e3cd9718a14159bc724cc60e0"
server_key_iv: "bc30109831e6c98826bc64519bb8d503"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.459442 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:18.499217 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.499485 15724 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260925 07:22:18.499585 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "a29f0487e06046eea676ef10c453bf02"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "dd5b595e3cd9718a14159bc724cc60e0"
server_key_iv: "bc30109831e6c98826bc64519bb8d503"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.499683 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:22:18.522909 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:22:18.525408 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:22:18.525555 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:22:18.525821 15724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:22:18.526235 15724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:22:18.526292 15724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.526358 15724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:22:18.526389 15724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.533435 15724 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:41941
I20260925 07:22:18.533903 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:22:18.534544 15855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:41941 every 8 connection(s)
I20260925 07:22:18.539479 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 15724
I20260925 07:22:18.539585 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:22:18.539763 4284 external_mini_cluster.cc:1428] Setting key f771737416f35ba03e3fb1ed0ee64aca
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:22:18.545850 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.534676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41173 (local address 127.4.47.2:55237)
0925 07:22:18.534837 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.534894 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.535715 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.536097 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.536102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.536337 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:22:18.536607 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.536614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.537606 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.537607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.538904 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.538907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.539505 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.539506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.539654 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.540293 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:22:18.540318 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:22:18.541950 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:22:18.544091 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.544097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.544100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.544443 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.544662 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.544663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.544663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.544788 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.545097 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:22:18.545104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:22:18.545287 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.545420 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.545586 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260925 07:22:18.546324 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.534976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55237 (local address 127.4.47.62:41173)
0925 07:22:18.535030 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.535032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.535128 (+ 96us) server_negotiation.cc:407] Connection header received
0925 07:22:18.535846 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.535848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.535861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.535906 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:22:18.536721 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.537535 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.539012 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.539364 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.539436 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.542063 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:22:18.542065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:22:18.542065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:22:18.542073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:22:18.543990 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.544537 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.544538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.544539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.544599 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.544859 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.544860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.544861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.545001 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:22:18.545066 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.546135 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.546231 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:22:18.546708 15856 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41173
I20260925 07:22:18.546854 15856 heartbeater.cc:464] Registering TS with master...
I20260925 07:22:18.547139 15856 heartbeater.cc:511] Master 127.4.47.62:41173 requested a full tablet report, sending...
I20260925 07:22:18.547629 15506 ts_manager.cc:194] Registered new tserver with Master: a29f0487e06046eea676ef10c453bf02 (127.4.47.2:41941)
I20260925 07:22:18.548213 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:55237
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:22:18.573750 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:47961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:22:18.665650 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:22:18.665884 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:22:18.665923 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:22:18.667169 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:22:18.667227 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:22:18.667271 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:22:18.668735 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.47.20:47961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:22:18.668987 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:22:18.669253 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:22:18.674291 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.674258 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:22:18.674536 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:22:18.675218 15861 server_base.cc:1034] running on GCE node
I20260925 07:22:18.675437 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:22:18.675700 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:22:18.676942 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790320938676916 us; error 38 us; skew 500 ppm
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:18.680143 15861 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:22:18.680707 15861 webserver.cc:460] Webserver started at http://127.4.47.3:34521/ using document root <none> and password file <none>
I20260925 07:22:18.680984 15861 fs_manager.cc:359] Metadata directory not provided
I20260925 07:22:18.681041 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:22:18.681181 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:18.696368 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "c4964dc7cfa54581858f18af0705d99e"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "5a2ef0025ec96b4beddf459783bd493b"
server_key_iv: "d5bb003964f17504d2a1d3e28784acd8"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.696851 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "c4964dc7cfa54581858f18af0705d99e"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "5a2ef0025ec96b4beddf459783bd493b"
server_key_iv: "d5bb003964f17504d2a1d3e28784acd8"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.699160 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 07:22:18.721438 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.721638 15861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 07:22:18.721738 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "c4964dc7cfa54581858f18af0705d99e"
format_stamp: "Formatted at 2026-09-25 07:22:18 on dist-test-slave-zw07"
server_key: "5a2ef0025ec96b4beddf459783bd493b"
server_key_iv: "d5bb003964f17504d2a1d3e28784acd8"
server_key_version: "kuduclusterkey@0"
I20260925 07:22:18.721886 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:22:18.747244 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:22:18.749339 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:22:18.749504 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:22:18.749752 15861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:22:18.750144 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:22:18.750196 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.750234 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:22:18.750270 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:22:18.757490 15861 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:45579
I20260925 07:22:18.757952 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:22:18.758800 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:45579 every 8 connection(s)
I20260925 07:22:18.763562 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 15861
I20260925 07:22:18.763671 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:22:18.763862 4284 external_mini_cluster.cc:1428] Setting key 7004da2874e34161c7f56fbda9976311
Sep 25 07:22:18 dist-test-slave-zw07 krb5kdc[15030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320938, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:22:18.777038 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.759807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:43095 (local address 127.4.47.62:41173)
0925 07:22:18.759856 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.759858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.760221 (+ 363us) server_negotiation.cc:407] Connection header received
0925 07:22:18.760894 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.760895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.760908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.760965 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:22:18.761590 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.762107 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.763169 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.763407 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.763475 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.766037 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:22:18.766038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:22:18.766038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:22:18.766046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:22:18.767891 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.775984 (+ 8093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.775985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.775986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.776061 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.776332 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.776333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.776334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.776460 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:22:18.776537 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.776853 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.776945 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:22:18.777557 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.759502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:41173 (local address 127.4.47.3:43095)
0925 07:22:18.760078 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.760134 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.760765 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.761083 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.761087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.761285 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:22:18.761485 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.761489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.762203 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.762204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.763019 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.763022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.763576 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.763577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.763737 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.764372 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:22:18.764382 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:22:18.765935 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:22:18.775136 (+ 9201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.775152 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.775155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.775801 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.776137 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:22:18.776138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:22:18.776138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:22:18.776259 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:22:18.776538 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:22:18.776546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:22:18.776719 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.776833 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.776999 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":84,"threads_started":1}
I20260925 07:22:18.778033 15990 heartbeater.cc:347] Connected to a master server at 127.4.47.62:41173
I20260925 07:22:18.778156 15990 heartbeater.cc:464] Registering TS with master...
I20260925 07:22:18.778371 15990 heartbeater.cc:511] Master 127.4.47.62:41173 requested a full tablet report, sending...
I20260925 07:22:18.778797 15506 ts_manager.cc:194] Registered new tserver with Master: c4964dc7cfa54581858f18af0705d99e (127.4.47.3:45579)
I20260925 07:22:18.779313 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:43095
I20260925 07:22:18.786834 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:22:18.801182 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.792571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.4.47.62:41173)
0925 07:22:18.792617 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.792620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.793006 (+ 386us) server_negotiation.cc:407] Connection header received
0925 07:22:18.793118 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.793119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.793131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.793164 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:22:18.793979 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.794738 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.795926 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.796203 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.796251 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.797920 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:22:18.797922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:22:18.797922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:22:18.797930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:22:18.799802 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.800211 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.800212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.800213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.800266 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:22:18.800471 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:22:18.800472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:22:18.800472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:22:18.800601 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:22:18.800679 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.801002 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.801084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:22:18.803380 15506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:22:18.805850 15506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:22:18.825729 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.812619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41941 (local address 127.0.0.1:56302)
0925 07:22:18.814536 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.814545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.814579 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.816609 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.816610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.816613 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:22:18.816667 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.816669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.822239 (+ 5570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.822240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.823373 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.823376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.825371 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.825372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.825474 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.825496 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.825574 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.825636 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1900,"thread_start_us":89,"threads_started":1}
I20260925 07:22:18.826506 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.812615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:45579 (local address 127.0.0.1:57352)
0925 07:22:18.814301 (+ 1686us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.814352 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.814424 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.815141 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.815145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.815148 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:22:18.815233 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.815234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.818009 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.818010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.819384 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.819386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.826289 (+ 6903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.826290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.826358 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.826362 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.826393 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.826435 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":88,"threads_started":1}
I20260925 07:22:18.827239 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.813103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.4.47.3:45579)
0925 07:22:18.814937 (+ 1834us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.814941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.814950 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:22:18.814985 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.814987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.815002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.815307 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:22:18.815386 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.816487 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.823505 (+ 7018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.824321 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.826982 (+ 2661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.827091 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.827121 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.827166 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":81,"threads_started":1}
I20260925 07:22:18.827657 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.812906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.4.47.2:41941)
0925 07:22:18.814700 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.814705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.814717 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:22:18.814764 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.814766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.814782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.816746 (+ 1964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:22:18.816862 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.817895 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.824464 (+ 6569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.825267 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.826831 (+ 1564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.827507 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.827537 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.827581 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1754,"thread_start_us":79,"threads_started":1}
I20260925 07:22:18.829689 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 (DEFAULT_TABLE table=test-table [id=e2c63c0675824dcc9b6f1fe4f83acb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:22:18.830091 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4923c769ca44463bb002f65b63f6bbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:22:18.834484 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.813179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.4.47.1:45331)
0925 07:22:18.814217 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.814223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.819451 (+ 5228us) server_negotiation.cc:407] Connection header received
0925 07:22:18.819493 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.819496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.819512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.819584 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:22:18.819787 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.820868 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.827892 (+ 7024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.828694 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.828747 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.828801 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.834345 (+ 5544us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.834411 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":59,"threads_started":1}
I20260925 07:22:18.834689 15789 tablet_service.cc:1467] Processing CreateTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 (DEFAULT_TABLE table=test-table [id=e2c63c0675824dcc9b6f1fe4f83acb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:22:18.835047 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4923c769ca44463bb002f65b63f6bbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:22:18.835086 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.812888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45331 (local address 127.0.0.1:51312)
0925 07:22:18.819151 (+ 6263us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.819163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.819223 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.819656 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.819657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.819658 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:22:18.819707 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.819708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.820976 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.820976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.822124 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.822127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.828878 (+ 6751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.828879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.828961 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.828964 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.834923 (+ 5959us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.834986 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6243,"thread_start_us":70,"threads_started":1}
I20260925 07:22:18.835911 15650 tablet_service.cc:1467] Processing CreateTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 (DEFAULT_TABLE table=test-table [id=e2c63c0675824dcc9b6f1fe4f83acb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:22:18.836225 15650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4923c769ca44463bb002f65b63f6bbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:22:18.839778 16008 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Bootstrap starting.
I20260925 07:22:18.840932 16008 tablet_bootstrap.cc:654] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Neither blocks nor log segments found. Creating new log.
I20260925 07:22:18.841135 16010 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: Bootstrap starting.
I20260925 07:22:18.841928 16010 tablet_bootstrap.cc:654] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: Neither blocks nor log segments found. Creating new log.
I20260925 07:22:18.842314 16010 log.cc:824] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:22:18.842752 16008 log.cc:824] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:22:18.844064 16010 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: No bootstrap required, opened a new log
I20260925 07:22:18.844221 16010 ts_tablet_manager.cc:1397] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:22:18.847352 16010 raft_consensus.cc:348] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.847709 16010 raft_consensus.cc:374] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:22:18.847815 16010 raft_consensus.cc:729] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfdcf08988a4af181cff05e36afa9a4, State: Initialized, Role: FOLLOWER
I20260925 07:22:18.848093 16010 consensus_queue.cc:260] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.848492 16010 ts_tablet_manager.cc:1428] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 07:22:18.848685 15716 heartbeater.cc:503] Master 127.4.47.62:41173 was elected leader, sending a full tablet report...
I20260925 07:22:18.850322 16008 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: No bootstrap required, opened a new log
I20260925 07:22:18.850469 16008 ts_tablet_manager.cc:1397] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260925 07:22:18.853039 16008 raft_consensus.cc:348] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.853298 16008 raft_consensus.cc:374] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:22:18.853400 16008 raft_consensus.cc:729] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4964dc7cfa54581858f18af0705d99e, State: Initialized, Role: FOLLOWER
I20260925 07:22:18.853643 16008 consensus_queue.cc:260] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.854058 16008 ts_tablet_manager.cc:1428] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:22:18.854238 15990 heartbeater.cc:503] Master 127.4.47.62:41173 was elected leader, sending a full tablet report...
I20260925 07:22:18.855257 16009 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: Bootstrap starting.
I20260925 07:22:18.856434 16009 tablet_bootstrap.cc:654] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: Neither blocks nor log segments found. Creating new log.
I20260925 07:22:18.856760 16009 log.cc:824] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: Log is configured to *not* fsync() on all Append() calls
I20260925 07:22:18.857363 16013 raft_consensus.cc:482] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:22:18.857478 16013 raft_consensus.cc:504] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.857808 16013 leader_election.cc:288] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29f0487e06046eea676ef10c453bf02 (127.4.47.2:41941), c4964dc7cfa54581858f18af0705d99e (127.4.47.3:45579)
I20260925 07:22:18.859050 16009 tablet_bootstrap.cc:492] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: No bootstrap required, opened a new log
I20260925 07:22:18.859227 16009 ts_tablet_manager.cc:1397] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 07:22:18.861326 16009 raft_consensus.cc:348] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.861554 16009 raft_consensus.cc:374] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:22:18.861644 16009 raft_consensus.cc:729] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29f0487e06046eea676ef10c453bf02, State: Initialized, Role: FOLLOWER
I20260925 07:22:18.861860 16009 consensus_queue.cc:260] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.862341 16009 ts_tablet_manager.cc:1428] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:22:18.862907 15856 heartbeater.cc:503] Master 127.4.47.62:41173 was elected leader, sending a full tablet report...
I20260925 07:22:18.869352 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.858200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:60943 (local address 127.4.47.3:45579)
0925 07:22:18.858239 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.858242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.862138 (+ 3896us) server_negotiation.cc:407] Connection header received
0925 07:22:18.862544 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.862545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.862557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.862589 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:22:18.862938 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.863683 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.868415 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.868950 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.869003 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.869014 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.869235 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.869278 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:22:18.869803 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.858025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:45579 (local address 127.4.47.1:60943)
0925 07:22:18.862065 (+ 4040us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.862077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.862444 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.862659 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.862660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.862661 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:22:18.862744 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.862746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.864290 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.864291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.865546 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.865550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.869078 (+ 3528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.869078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.869149 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.869152 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.869657 (+ 505us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.869705 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4019,"thread_start_us":80,"threads_started":1}
I20260925 07:22:18.870321 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4923c769ca44463bb002f65b63f6bbf1" candidate_uuid: "4bfdcf08988a4af181cff05e36afa9a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4964dc7cfa54581858f18af0705d99e" is_pre_election: true
I20260925 07:22:18.870502 15944 raft_consensus.cc:2454] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfdcf08988a4af181cff05e36afa9a4 in term 0.
I20260925 07:22:18.871019 15603 leader_election.cc:302] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfdcf08988a4af181cff05e36afa9a4, c4964dc7cfa54581858f18af0705d99e; no voters:
I20260925 07:22:18.871183 16013 raft_consensus.cc:2781] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:22:18.871229 16013 raft_consensus.cc:482] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:22:18.871254 16013 raft_consensus.cc:3037] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:22:18.872193 16013 raft_consensus.cc:504] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.872331 16013 leader_election.cc:288] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [CANDIDATE]: Term 1 election: Requested vote from peers a29f0487e06046eea676ef10c453bf02 (127.4.47.2:41941), c4964dc7cfa54581858f18af0705d99e (127.4.47.3:45579)
I20260925 07:22:18.872671 15944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4923c769ca44463bb002f65b63f6bbf1" candidate_uuid: "4bfdcf08988a4af181cff05e36afa9a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4964dc7cfa54581858f18af0705d99e"
I20260925 07:22:18.872749 15944 raft_consensus.cc:3037] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:22:18.873622 15944 raft_consensus.cc:2454] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfdcf08988a4af181cff05e36afa9a4 in term 1.
I20260925 07:22:18.873853 15603 leader_election.cc:302] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfdcf08988a4af181cff05e36afa9a4, c4964dc7cfa54581858f18af0705d99e; no voters:
I20260925 07:22:18.873988 16013 raft_consensus.cc:2781] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:22:18.874140 16013 raft_consensus.cc:686] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4bfdcf08988a4af181cff05e36afa9a4, State: Running, Role: LEADER
I20260925 07:22:18.874225 16013 consensus_queue.cc:237] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } }
I20260925 07:22:18.875104 15506 catalog_manager.cc:5495] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bfdcf08988a4af181cff05e36afa9a4 (127.4.47.1). New cstate: current_term: 1 leader_uuid: "4bfdcf08988a4af181cff05e36afa9a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bfdcf08988a4af181cff05e36afa9a4" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 45331 } health_report { overall_health: HEALTHY } } }
I20260925 07:22:18.877265 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.858640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:59065 (local address 127.4.47.2:41941)
0925 07:22:18.858678 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.858680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.865721 (+ 7041us) server_negotiation.cc:407] Connection header received
0925 07:22:18.865817 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.865818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.865834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.865863 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:22:18.866098 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.866771 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.876227 (+ 9456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.876835 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.876901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.876913 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.877153 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.877196 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:22:18.877655 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.858451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41941 (local address 127.4.47.1:59065)
0925 07:22:18.865664 (+ 7213us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.865675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.865746 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.865947 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:22:18.865948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:22:18.865949 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:22:18.866019 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.866020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.866906 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.866906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.868295 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:22:18.868298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.876982 (+ 8684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.876983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:22:18.877067 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.877071 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:22:18.877508 (+ 437us) client_negotiation.cc:240] Negotiation successful
0925 07:22:18.877559 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7194,"thread_start_us":102,"threads_started":1}
I20260925 07:22:18.878152 15810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4923c769ca44463bb002f65b63f6bbf1" candidate_uuid: "4bfdcf08988a4af181cff05e36afa9a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29f0487e06046eea676ef10c453bf02" is_pre_election: true
I20260925 07:22:18.878316 15810 raft_consensus.cc:2454] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfdcf08988a4af181cff05e36afa9a4 in term 0.
I20260925 07:22:18.878465 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4923c769ca44463bb002f65b63f6bbf1" candidate_uuid: "4bfdcf08988a4af181cff05e36afa9a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29f0487e06046eea676ef10c453bf02"
I20260925 07:22:18.878547 15809 raft_consensus.cc:3037] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:22:18.879477 15809 raft_consensus.cc:2454] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfdcf08988a4af181cff05e36afa9a4 in term 1.
I20260925 07:22:18.903506 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.900025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.4.47.1:45331)
0925 07:22:18.900066 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:22:18.900067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:22:18.900075 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:22:18.900123 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:22:18.900124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:22:18.900147 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:22:18.900215 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:22:18.900402 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.901111 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.902249 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:22:18.902538 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:22:18.902609 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:22:18.902852 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:22:18.903011 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:22:18.903170 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 07:22:18.903368 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 07:22:18.903391 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:22:18.905714 15944 raft_consensus.cc:1261] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 1 FOLLOWER]: Refusing update from remote peer 4bfdcf08988a4af181cff05e36afa9a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:22:18.905786 15809 raft_consensus.cc:1261] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 1 FOLLOWER]: Refusing update from remote peer 4bfdcf08988a4af181cff05e36afa9a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:22:18.906054 16019 consensus_queue.cc:1035] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4964dc7cfa54581858f18af0705d99e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 45579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:22:18.906203 16013 consensus_queue.cc:1035] T 4923c769ca44463bb002f65b63f6bbf1 P 4bfdcf08988a4af181cff05e36afa9a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29f0487e06046eea676ef10c453bf02" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:22:18.916254 16023 mvcc.cc:205] Tried to move back new op lower bound from 7333154565754720256 to 7333154565629444096. Current Snapshot: MvccSnapshot[applied={T|T < 7333154565754720256}]
I20260925 07:22:18.917367 16024 mvcc.cc:205] Tried to move back new op lower bound from 7333154565754720256 to 7333154565629444096. Current Snapshot: MvccSnapshot[applied={T|T < 7333154565754720256}]
I20260925 07:22:18.917783 16022 mvcc.cc:205] Tried to move back new op lower bound from 7333154565754720256 to 7333154565629444096. Current Snapshot: MvccSnapshot[applied={T|T < 7333154565754720256}]
I20260925 07:22:18.922029 15505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:22:18.922127 15505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:22:18.922946 15505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec5f09d0d1f74b879f67238bcfeb8d67: Sending DeleteTablet for 3 replicas of tablet 4923c769ca44463bb002f65b63f6bbf1
I20260925 07:22:18.923382 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:22:18 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:56302
I20260925 07:22:18.923610 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:22:18 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:57352
I20260925 07:22:18.924072 15650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4923c769ca44463bb002f65b63f6bbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:22:18 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:51312
I20260925 07:22:18.924683 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 15587
I20260925 07:22:18.931612 16032 tablet_replica.cc:330] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: stopping tablet replica
I20260925 07:22:18.931778 16031 tablet_replica.cc:330] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02: stopping tablet replica
I20260925 07:22:18.932101 16032 raft_consensus.cc:2229] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:22:18.931996 16031 raft_consensus.cc:2229] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:22:18.932405 16032 raft_consensus.cc:2258] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:22:18.932570 16031 raft_consensus.cc:2258] T 4923c769ca44463bb002f65b63f6bbf1 P a29f0487e06046eea676ef10c453bf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:22:18.933390 16032 ts_tablet_manager.cc:1905] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:22:18.936868 16032 ts_tablet_manager.cc:1918] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:22:18.937026 16032 log.cc:1196] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/4923c769ca44463bb002f65b63f6bbf1
I20260925 07:22:18.937690 16032 ts_tablet_manager.cc:1939] T 4923c769ca44463bb002f65b63f6bbf1 P c4964dc7cfa54581858f18af0705d99e: Deleting consensus metadata
W20260925 07:22:18.938190 15491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:45331 (error 108)
I20260925 07:22:18.939183 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 15724
I20260925 07:22:18.940614 15490 catalog_manager.cc:4841] TS c4964dc7cfa54581858f18af0705d99e (127.4.47.3:45579): tablet 4923c769ca44463bb002f65b63f6bbf1 (table test-table [id=e2c63c0675824dcc9b6f1fe4f83acb62]) successfully deleted
I20260925 07:22:18.941155 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:22:18.941014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:45331 (local address 127.0.0.1:51338)
0925 07:22:18.941066 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.941121 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:45331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 07:22:18.941289 15491 catalog_manager.cc:4584] TS 4bfdcf08988a4af181cff05e36afa9a4 (127.4.47.1:45331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4923c769ca44463bb002f65b63f6bbf1: Network error: Client connection negotiation failed: client connection to 127.4.47.1:45331: connect: Connection refused (error 111)
W20260925 07:22:18.946007 16003 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:22:18.945755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41941 (local address 127.0.0.1:56308)
0925 07:22:18.945800 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:22:18.945811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:22:18.945850 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:22:18.945959 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:41941: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 07:22:18.951449 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 15861
I20260925 07:22:18.957320 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 15474
I20260925 07:22:18.964154 4284 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 07:22:19.309147 4284 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 07:22:19.309252 4284 mini_ranger.cc:66] Stopping Ranger...
I20260925 07:22:19.653622 4284 mini_ranger.cc:68] Stopped Ranger
2026-09-25 07:22:19.653 UTC [15060] LOG: received smart shutdown request
2026-09-25 07:22:19.657 UTC [15060] LOG: background worker "logical replication launcher" (PID 15069) exited with exit code 1
2026-09-25 07:22:19.657 UTC [15064] LOG: shutting down
2026-09-25 07:22:19.844 UTC [15060] LOG: database system is shut down
2026-09-25T07:22:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46835 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:22:20 dist-test-slave-zw07 krb5kdc[16043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:22:20 dist-test-slave-zw07 krb5kdc[16043](info): set up 2 sockets
Sep 25 07:22:20 dist-test-slave-zw07 krb5kdc[16043](info): commencing operation
krb5kdc: starting...
W20260925 07:22:20.994951 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:22:21 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:22:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangeradmin_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangeradmin_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangerlookup_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangerlookup_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
I20260925 07:22:21.162331 4284 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSRTLta/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/postgres -l logfile start
2026-09-25 07:22:23.707 UTC [16073] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 07:22:23.707 UTC [16073] LOG: listening on IPv4 address "127.4.47.20", port 42647
2026-09-25 07:22:23.711 UTC [16073] LOG: listening on Unix socket "/tmp/.s.PGSQL.42647"
2026-09-25 07:22:23.748 UTC [16076] LOG: database system was shut down at 2026-09-25 07:22:22 UTC
2026-09-25 07:22:23.752 UTC [16073] LOG: database system is ready to accept connections
I20260925 07:22:24.710826 4284 mini_postgres.cc:96] Postgres bound to 42647
2026-09-25 07:22:24.716 UTC [16084] FATAL: database "slave" does not exist
127.4.47.20:42647 - accepting connections
I20260925 07:22:24.717231 4284 mini_ranger.cc:161] Starting Ranger...
I20260925 07:22:24.731802 4284 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 07:22:25.239526 4284 mini_ranger.cc:87] Created ranger Postgres database
I20260925 07:22:25.239621 4284 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin
2026-09-25 07:22:25,527 [I] DB FLAVOR :POSTGRES
2026-09-25 07:22:25,528 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:22:25,528 [I] Checking connection..
2026-09-25 07:22:25,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 07:22:25,790 [I] Checking connection passed.
2026-09-25 07:22:25,790 [I] --------- Verifying version history table ---------
2026-09-25 07:22:25,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:22:26,060 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:22:26,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:22:26,313 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:22:26,314 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 07:22:26,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 07:22:26.559 UTC [16148] WARNING: there is no transaction in progress
2026-09-25 07:22:26,566 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 07:22:26,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:22:26,830 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 07:22:26,830 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:22:26,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 07:22:27,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 07:22:27,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zw07.c.gcp-upstream.internal','N') ;"
2026-09-25 07:22:27,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:22:27,772 [I] Table x_portal_user does not exist in database ranger
2026-09-25 07:22:27,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:22:28,026 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 07:22:28,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:22:28,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:22:28,508 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 07:22:28,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 07:22:29.513 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.529 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.546 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.562 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.579 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.597 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.663 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.671 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.688 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.699 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.710 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.725 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.734 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.741 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.748 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:29.755 UTC [16256] WARNING: there is no transaction in progress
2026-09-25 07:22:30,411 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 07:22:30,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 07:22:30,653 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 07:22:30,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:22:30,914 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 07:22:30,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:22:31,149 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 07:22:31,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:22:31,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:22:31,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zw07.c.gcp-upstream.internal';"
2026-09-25 07:22:31,876 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 07:22:31,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 07:22:32,107 [I] DB_PATCHES have already been applied
I20260925 07:22:32.113081 4284 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSRTLta/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 07:22:32.113183 4284 mini_ranger.cc:193] Using host: 127.4.47.20
I20260925 07:22:32.116344 4284 mini_ranger.cc:227] Ranger admin URL: http://127.4.47.20:47239
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:22:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 07:22:33.224555 4284 mini_ranger.cc:160] Time spent starting Ranger: real 8.507s user 0.000s sys 0.009s
Sep 25, 2026 7:22:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:22:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 7:22:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:22:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:22:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 07:22:54.048977 4284 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.47.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.47.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangerkms_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.47.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/rangerkms_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.47.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.47.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.47.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/HTTP_127.4.47.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 07:22:54 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 07:22:54.138666 4284 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 07:22:54.149178 4284 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 07:22:55.090673 4284 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 07:22:55.090770 4284 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms
I20260925 07:22:55.090788 4284 mini_ranger_kms.cc:227] Using postgres at 127.4.47.20:42647
2026-09-25 07:22:55,257 [I] DB FLAVOR :POSTGRES
2026-09-25 07:22:55,257 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:22:55,257 [I] Checking connection
2026-09-25 07:22:55,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 07:22:55,489 [I] connection success
2026-09-25 07:22:55,490 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 07:22:55,490 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 07:22:55,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 07:22:55,725 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 07:22:55,725 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:22:55,725 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 07:22:55,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.47.20:42647/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 07:22:55,979 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 07:22:56.465129 4284 mini_ranger_kms.cc:310] Created kms service
I20260925 07:22:56.656742 4284 mini_ranger_kms.cc:326] Created kudu user
I20260925 07:22:56.701517 4284 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 07:22:56 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:22:57.106366 4284 mini_ranger_kms.cc:384] Added ranger policy
I20260925 07:22:57.106483 4284 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSRTLta/build/release/bin/postgresql.jar:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSRTLta/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 07:22:57.106503 4284 mini_ranger_kms.cc:240] Using host: 127.4.47.20
I20260925 07:22:57.109997 4284 mini_ranger_kms.cc:276] Ranger KMS PID: 16436
I20260925 07:22:57.110085 4284 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.47.20:53589
Sep 25, 2026 7:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:22:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:22:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 07:22:59 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790320979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:22:59.399524 4284 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.261s user 0.000s sys 0.147s
I20260925 07:22:59.399627 4284 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 07:22:59.399681 4284 mini_ranger_kms.cc:401] 127.4.47.20:53589/kms/v1/keys
Sep 25 07:22:59 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
Sep 25 07:23:00 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.47.20@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
Sep 25 07:23:02 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:23:02.304291 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:39617
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:39617
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:02.387148 16487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:02.387363 16487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:02.387384 16487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:02.388564 16487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:02.388615 16487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:02.388630 16487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:02.388643 16487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:02.388654 16487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:02.390326 16487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.47.20:53589/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:39617
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:02.390574 16487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:02.390825 16487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:02.397099 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:02.397158 16487 server_base.cc:1034] running on GCE node
W20260925 07:23:02.397203 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:02.397110 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:02.397799 16487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:02.398157 16487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:02.399334 16487 hybrid_clock.cc:648] HybridClock initialized: now 1790320982399310 us; error 37 us; skew 500 ppm
Sep 25 07:23:02 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:02.402335 16487 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:02.402808 16487 webserver.cc:460] Webserver started at http://127.4.47.62:41159/ using document root <none> and password file <none>
I20260925 07:23:02.403045 16487 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:02.403127 16487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:02.403301 16487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:23:02 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:02.450503 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "c6b07ab5444b4066b8c76e2fa29e2434"
format_stamp: "Formatted at 2026-09-25 07:23:02 on dist-test-slave-zw07"
server_key: "d2e3766bc64ea8537a5d33c8510be0a6"
server_key_iv: "edb6fcf52d04c1cd8f7abf2a9bb72904"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:02.450938 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "c6b07ab5444b4066b8c76e2fa29e2434"
format_stamp: "Formatted at 2026-09-25 07:23:02 on dist-test-slave-zw07"
server_key: "d2e3766bc64ea8537a5d33c8510be0a6"
server_key_iv: "edb6fcf52d04c1cd8f7abf2a9bb72904"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:02.452450 16487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:02.464596 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:02.464839 16487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:02.464951 16487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c6b07ab5444b4066b8c76e2fa29e2434"
format_stamp: "Formatted at 2026-09-25 07:23:02 on dist-test-slave-zw07"
server_key: "d2e3766bc64ea8537a5d33c8510be0a6"
server_key_iv: "edb6fcf52d04c1cd8f7abf2a9bb72904"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:02.465070 16487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:02.474653 16487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:02.476881 16487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:02.477061 16487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:02.481675 16487 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:39617
I20260925 07:23:02.481711 16554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:39617 every 8 connection(s)
I20260925 07:23:02.482088 16487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:02.482796 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:02.485172 16555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Bootstrap starting.
I20260925 07:23:02.485792 16555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:02.486099 16555 log.cc:824] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:02.486789 16555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: No bootstrap required, opened a new log
I20260925 07:23:02.488876 16555 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:02.489022 16555 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:02.489044 16555 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b07ab5444b4066b8c76e2fa29e2434, State: Initialized, Role: FOLLOWER
I20260925 07:23:02.489194 16555 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:02.489282 16555 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:02.489324 16555 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:02.489373 16555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:02.489904 16555 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:02.489992 16555 leader_election.cc:302] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b07ab5444b4066b8c76e2fa29e2434; no voters:
I20260925 07:23:02.490170 16555 leader_election.cc:288] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:23:02.490207 16558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:02.490401 16558 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 LEADER]: Becoming Leader. State: Replica: c6b07ab5444b4066b8c76e2fa29e2434, State: Running, Role: LEADER
I20260925 07:23:02.490476 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 16487
I20260925 07:23:02.490576 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:23:02.490546 16558 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:02.490765 4284 external_mini_cluster.cc:1428] Setting key f8c95c41ec648279507719e27b21ca8c
I20260925 07:23:02.490908 16558 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b07ab5444b4066b8c76e2fa29e2434. Latest consensus state: current_term: 1 leader_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } } }
I20260925 07:23:02.490978 16558 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:02.491015 16559 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } } }
I20260925 07:23:02.491111 16559 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:02.491963 16555 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:02.492048 16562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:02.492555 16562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:02.493539 16555 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 07:23:02.495044 16562 catalog_manager.cc:1347] Generated new cluster ID: 1cb19ee5e4664511ad8ed852a2461aa0
I20260925 07:23:02.495131 16562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:23:02 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:02.501767 16562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:23:02.502668 16562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:02.503741 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:02.491524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.4.47.62:39617)
0925 07:23:02.491860 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:02.491871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:02.491903 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:23:02.492024 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:02.492032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:02.492093 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:02.493130 (+ 1037us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:02.493259 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:02.494492 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:02.495601 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:02.496201 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:02.496311 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:02.499349 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:02.499355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:02.499355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:02.499371 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:02.501472 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:02.501943 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:02.501946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:02.501947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:02.502061 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:02.502297 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:02.502298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:02.502298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:02.502673 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:02.502815 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:02.503378 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 07:23:02.503510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":290,"thread_start_us":70,"threads_started":1}
I20260925 07:23:02.513069 16562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Generated new TSK 0
I20260925 07:23:02.513499 16562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:23:03.270838 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:03.390225 16601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:03.390516 16601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:03.390555 16601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:03.392422 16601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:03.392493 16601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:03.392534 16601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:03.394629 16601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:03.395051 16601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:03.395371 16601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:03.402076 16610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.402416 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.402577 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:03.403775 16601 server_base.cc:1034] running on GCE node
I20260925 07:23:03.404026 16601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:03.404338 16601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:03.405524 16601 hybrid_clock.cc:648] HybridClock initialized: now 1790320983405500 us; error 35 us; skew 500 ppm
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:03.408947 16601 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:03.409531 16601 webserver.cc:460] Webserver started at http://127.4.47.1:41641/ using document root <none> and password file <none>
I20260925 07:23:03.409812 16601 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:03.409893 16601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:03.410038 16601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:03.429109 16601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "1dc05234cca64039af5502508a168a6f"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b7af4d10d9bcb325c50a2ef9359c7542"
server_key_iv: "cead7aa686725d0b296f073d42a9fff6"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.429713 16601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "1dc05234cca64039af5502508a168a6f"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b7af4d10d9bcb325c50a2ef9359c7542"
server_key_iv: "cead7aa686725d0b296f073d42a9fff6"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.432447 16601 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 07:23:03.450255 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.450505 16601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:03.450601 16601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "1dc05234cca64039af5502508a168a6f"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b7af4d10d9bcb325c50a2ef9359c7542"
server_key_iv: "cead7aa686725d0b296f073d42a9fff6"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.450698 16601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:03.463160 16601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:03.465794 16601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:03.466027 16601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:03.466415 16601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:03.466890 16601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:03.466987 16601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.467123 16601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:03.467242 16601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.496762 16601 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:33539
I20260925 07:23:03.497376 16601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:03.497972 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 16601
I20260925 07:23:03.498086 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:23:03.498319 4284 external_mini_cluster.cc:1428] Setting key 9d85673af396990fef2004d31fb65f68
I20260925 07:23:03.502619 16731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:33539 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
I20260925 07:23:03.526150 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:03.503394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.1:53761)
0925 07:23:03.503857 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:03.503913 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:03.504576 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:03.505039 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:03.505044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:03.505316 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:03.505627 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:03.505634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.506721 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.506722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:03.508034 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:03.508038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.508616 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.508617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:03.508773 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:03.509424 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:03.509436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:03.514474 (+ 5038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:03.524231 (+ 9757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:03.524246 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:03.524249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:03.524611 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:03.524867 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:03.524868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:03.524868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:03.525023 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:03.525327 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:03.525336 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:03.525565 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 07:23:03.525673 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:23:03.525885 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":92,"threads_started":1}
I20260925 07:23:03.526569 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:03.503601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:53761 (local address 127.4.47.62:39617)
0925 07:23:03.504803 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:03.504805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:03.504814 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:03.504840 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:03.504841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:03.504854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:03.505737 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:03.505799 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.506606 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.508156 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.508455 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.508524 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:03.514614 (+ 6090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:03.514617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:03.514617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:03.514628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:03.524084 (+ 9456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:03.524721 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:03.524722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:03.524722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:03.524794 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:03.525099 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:03.525099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:03.525100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:03.525247 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:03.526324 (+ 1077us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:03.526369 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 07:23:03.526466 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":120,"threads_started":1}
I20260925 07:23:03.527101 16732 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:03.527209 16732 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:03.527410 16732 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:03.528005 16519 ts_manager.cc:194] Registered new tserver with Master: 1dc05234cca64039af5502508a168a6f (127.4.47.1:33539)
I20260925 07:23:03.528925 16519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:53761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:23:03.548975 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 07:23:03.678030 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:03.678404 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:03.678480 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:03.680521 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:03.680588 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:03.680629 16740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:03.682845 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:03.683269 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:03.683530 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:03.689687 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.689904 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.689682 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:03.690153 16740 server_base.cc:1034] running on GCE node
I20260925 07:23:03.690372 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:03.690644 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:03.691821 16740 hybrid_clock.cc:648] HybridClock initialized: now 1790320983691801 us; error 44 us; skew 500 ppm
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:03.695276 16740 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:03.695840 16740 webserver.cc:460] Webserver started at http://127.4.47.2:41413/ using document root <none> and password file <none>
I20260925 07:23:03.696215 16740 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:03.696288 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:03.696439 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:03.709342 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "db9e4875ebee4ae68a4acdd9bd65ad38"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b4fab308f89a1d4418345330f01e5f19"
server_key_iv: "37994f838e6380d5a08c365299607ae8"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.709793 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "db9e4875ebee4ae68a4acdd9bd65ad38"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b4fab308f89a1d4418345330f01e5f19"
server_key_iv: "37994f838e6380d5a08c365299607ae8"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.718606 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260925 07:23:03.741405 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.741667 16740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:03.741760 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "db9e4875ebee4ae68a4acdd9bd65ad38"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b4fab308f89a1d4418345330f01e5f19"
server_key_iv: "37994f838e6380d5a08c365299607ae8"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:03.741852 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:03.784300 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:03.786918 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:03.787176 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:03.787525 16740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:03.787976 16740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:03.788074 16740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.788160 16740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:03.788233 16740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:03.794669 16740 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:37965
I20260925 07:23:03.795190 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:23:03.798008 16869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:37965 every 8 connection(s)
I20260925 07:23:03.803268 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 16740
I20260925 07:23:03.803375 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:23:03.803587 4284 external_mini_cluster.cc:1428] Setting key 9ed09922d2b0376e321e791ada347533
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:03.809638 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:03.798644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:58161 (local address 127.4.47.62:39617)
0925 07:23:03.798696 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:03.798699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:03.798998 (+ 299us) server_negotiation.cc:407] Connection header received
0925 07:23:03.799739 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:03.799740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:03.799754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:03.799786 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:03.800641 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.801462 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.802944 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.803350 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.803406 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:03.806137 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:03.806139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:03.806139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:03.806147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:03.807932 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:03.808437 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:03.808439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:03.808439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:03.808500 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:03.808788 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:03.808789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:03.808789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:03.808940 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:03.808985 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:03.809420 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 07:23:03.809514 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:23:03.810485 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:03.798645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.2:58161)
0925 07:23:03.798869 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:03.798911 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:03.799608 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:03.799978 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:03.799982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:03.800244 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:03.800523 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:03.800530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.801586 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.801587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:03.802837 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:03.802840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:03.803525 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:03.803526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:03.803672 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:03.804301 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:03.804312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:03.806030 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:03.808030 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:03.808037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:03.808039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:03.808344 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:03.808570 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:03.808571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:03.808571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:03.808703 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:03.809034 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:03.809040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:03.809244 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 07:23:03.810046 (+ 802us) client_negotiation.cc:240] Negotiation successful
0925 07:23:03.810239 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260925 07:23:03.811282 16870 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:03.811386 16870 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:03.811590 16870 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:03.812041 16519 ts_manager.cc:194] Registered new tserver with Master: db9e4875ebee4ae68a4acdd9bd65ad38 (127.4.47.2:37965)
I20260925 07:23:03.812556 16519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:58161
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:23:03.841892 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:03.977483 16875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:03.977864 16875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:03.977952 16875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:03.979720 16875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:03.979966 16875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:03.980057 16875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:03.982256 16875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:03.982755 16875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:03.983114 16875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:03.989606 16881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.989856 16880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:03.989930 16883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:03.990978 16875 server_base.cc:1034] running on GCE node
I20260925 07:23:03.991328 16875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:03.991601 16875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:03.996553 16875 hybrid_clock.cc:648] HybridClock initialized: now 1790320983995934 us; error 633 us; skew 500 ppm
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:03.999917 16875 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:04.000495 16875 webserver.cc:460] Webserver started at http://127.4.47.3:42325/ using document root <none> and password file <none>
I20260925 07:23:04.000801 16875 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:04.000905 16875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:04.001075 16875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:23:03 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:04.025520 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "9a16cef8ed144575bc83fd398f7245e6"
format_stamp: "Formatted at 2026-09-25 07:23:04 on dist-test-slave-zw07"
server_key: "ccf8de1e2f7f8fb1807f86c474de4ca5"
server_key_iv: "91eeb4c3cbd169646b1a268e635e5364"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.026130 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "9a16cef8ed144575bc83fd398f7245e6"
format_stamp: "Formatted at 2026-09-25 07:23:04 on dist-test-slave-zw07"
server_key: "ccf8de1e2f7f8fb1807f86c474de4ca5"
server_key_iv: "91eeb4c3cbd169646b1a268e635e5364"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.031420 16875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260925 07:23:04.053615 16890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.053934 16875 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260925 07:23:04.054088 16875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "9a16cef8ed144575bc83fd398f7245e6"
format_stamp: "Formatted at 2026-09-25 07:23:04 on dist-test-slave-zw07"
server_key: "ccf8de1e2f7f8fb1807f86c474de4ca5"
server_key_iv: "91eeb4c3cbd169646b1a268e635e5364"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.054284 16875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:04.069525 16875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:04.073506 16875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:04.073668 16875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:04.073911 16875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:04.074270 16875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:04.074316 16875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.074355 16875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:04.074373 16875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.085750 16875 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:33761
I20260925 07:23:04.086194 16875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:04.089248 17003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:33761 every 8 connection(s)
I20260925 07:23:04.091259 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 16875
I20260925 07:23:04.091373 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:23:04.091539 4284 external_mini_cluster.cc:1428] Setting key e6d2f4340555a59baa55acee5ef4668f
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:04.102600 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.090151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:35455 (local address 127.4.47.62:39617)
0925 07:23:04.090194 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:04.090196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:04.090533 (+ 337us) server_negotiation.cc:407] Connection header received
0925 07:23:04.092656 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:04.092657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:04.092670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:04.092699 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:04.093553 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.094347 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.095845 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.096148 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.096187 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.098832 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:04.098833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:04.098834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:04.098842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:04.100845 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.101385 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.101386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.101386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.101453 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.101732 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.101732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.101733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.101882 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:04.101925 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:04.102383 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 07:23:04.102499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:23:04.104023 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.089868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.3:35455)
0925 07:23:04.090407 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:04.090453 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:04.092544 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:04.092891 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:04.092896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:04.093145 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:04.093445 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.093451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.094457 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.094458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.095745 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.095748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.096298 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.096299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.096438 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.097057 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:04.097068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:04.098721 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:04.100956 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.100964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.100967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.101294 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.101523 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.101523 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.101524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.101659 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.101971 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:04.101977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:04.102194 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 07:23:04.103597 (+ 1403us) client_negotiation.cc:240] Negotiation successful
0925 07:23:04.103777 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":88,"threads_started":1}
I20260925 07:23:04.104792 17004 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:04.104887 17004 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.105077 17004 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:04.105477 16519 ts_manager.cc:194] Registered new tserver with Master: 9a16cef8ed144575bc83fd398f7245e6 (127.4.47.3:33761)
I20260925 07:23:04.105975 16519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:35455
I20260925 07:23:04.114279 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:04.115018 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 16601
I20260925 07:23:04.120818 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 16740
I20260925 07:23:04.127843 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 16875
I20260925 07:23:04.137984 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 16487
I20260925 07:23:04.144418 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:39617
--webserver_interface=127.4.47.62
--webserver_port=41159
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:39617
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:04.228236 17007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:04.228456 17007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:04.228479 17007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:04.229743 17007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:04.229792 17007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:04.229806 17007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:04.229818 17007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:04.229831 17007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:04.231500 17007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:39617
--ranger_config_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=41159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:04.231740 17007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:04.231964 17007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:04.237301 17013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.237339 17012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.237301 17015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:04.237679 17007 server_base.cc:1034] running on GCE node
I20260925 07:23:04.237960 17007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:04.238251 17007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:04.239418 17007 hybrid_clock.cc:648] HybridClock initialized: now 1790320984239404 us; error 47 us; skew 500 ppm
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:04.241837 17007 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:04.242283 17007 webserver.cc:460] Webserver started at http://127.4.47.62:41159/ using document root <none> and password file <none>
I20260925 07:23:04.242523 17007 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:04.242599 17007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:04.310523 17007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:04.311367 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.311560 17007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:04.311658 17007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c6b07ab5444b4066b8c76e2fa29e2434"
format_stamp: "Formatted at 2026-09-25 07:23:02 on dist-test-slave-zw07"
server_key: "d2e3766bc64ea8537a5d33c8510be0a6"
server_key_iv: "edb6fcf52d04c1cd8f7abf2a9bb72904"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.312047 17007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:04.326504 17007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:04.328413 17007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:04.328583 17007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:04.333010 17007 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:39617
I20260925 07:23:04.333050 17075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:39617 every 8 connection(s)
I20260925 07:23:04.333434 17007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:04.334717 17076 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:23:04.335608 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Bootstrap starting.
I20260925 07:23:04.340690 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17007
I20260925 07:23:04.340986 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:33539
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:39617
--webserver_port=41641
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:04.343788 17076 log.cc:824] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:04.345499 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:04.345795 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Bootstrap complete.
I20260925 07:23:04.348308 17076 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:04.348579 17076 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b07ab5444b4066b8c76e2fa29e2434, State: Initialized, Role: FOLLOWER
I20260925 07:23:04.348770 17076 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:04.348851 17076 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:04.348939 17076 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:04.348995 17076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:23:04.350026 17076 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:04.350149 17076 leader_election.cc:302] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b07ab5444b4066b8c76e2fa29e2434; no voters:
I20260925 07:23:04.350374 17076 leader_election.cc:288] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:23:04.350414 17080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:23:04.350519 17080 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [term 2 LEADER]: Becoming Leader. State: Replica: c6b07ab5444b4066b8c76e2fa29e2434, State: Running, Role: LEADER
I20260925 07:23:04.350617 17080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } }
I20260925 07:23:04.350647 17076 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:04.351104 17082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b07ab5444b4066b8c76e2fa29e2434. Latest consensus state: current_term: 2 leader_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } } }
I20260925 07:23:04.351184 17082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:04.351425 17085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:04.351919 17085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:04.352032 17085 catalog_manager.cc:1259] Loaded cluster ID: 1cb19ee5e4664511ad8ed852a2461aa0
I20260925 07:23:04.352056 17085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:23:04.352293 17076 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 07:23:04.352867 17081 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b07ab5444b4066b8c76e2fa29e2434" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 39617 } } }
I20260925 07:23:04.353006 17081 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:04.353179 17085 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:04.353379 17085 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Loaded TSK: 0
I20260925 07:23:04.353595 17085 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:23:04.430903 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:04.431284 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:04.431386 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:04.433456 17078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:04.433573 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:04.433665 17078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:04.435943 17078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:33539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=41641
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:04.436205 17078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:04.436522 17078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:04.442863 17114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.442873 17113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.443181 17116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:04.443719 17078 server_base.cc:1034] running on GCE node
I20260925 07:23:04.443921 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:04.444198 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:04.445361 17078 hybrid_clock.cc:648] HybridClock initialized: now 1790320984445332 us; error 46 us; skew 500 ppm
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:04.448645 17078 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:04.449221 17078 webserver.cc:460] Webserver started at http://127.4.47.1:41641/ using document root <none> and password file <none>
I20260925 07:23:04.449497 17078 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:04.449579 17078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:04.471588 17078 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260925 07:23:04.475551 17124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.475797 17078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260925 07:23:04.475896 17078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "1dc05234cca64039af5502508a168a6f"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b7af4d10d9bcb325c50a2ef9359c7542"
server_key_iv: "cead7aa686725d0b296f073d42a9fff6"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.476413 17078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:04.517351 17078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:04.520820 17078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:04.521042 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:04.521296 17078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:04.521677 17078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:04.521754 17078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.521848 17078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:04.521914 17078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.528489 17078 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:33539
I20260925 07:23:04.529035 17078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:04.529394 17237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:33539 every 8 connection(s)
I20260925 07:23:04.532343 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17078
I20260925 07:23:04.532629 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:37965
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:39617
--webserver_port=41413
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:04.553853 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.530005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.1:59965)
0925 07:23:04.531193 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:04.531251 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:04.531893 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:04.533978 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:04.533983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:04.534246 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:04.534543 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.534550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.536363 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.536365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.537704 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.537708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.543656 (+ 5948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.543659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.543857 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.544549 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:04.544559 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:04.546155 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:04.551919 (+ 5764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.551931 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.551934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.552309 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.552597 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.552598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.552599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.552726 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.553142 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:04.553157 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:04.553306 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 07:23:04.553417 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 07:23:04.553604 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":113,"threads_started":1}
I20260925 07:23:04.553838 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.530199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:59965 (local address 127.4.47.62:39617)
0925 07:23:04.530418 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:04.530428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:04.531390 (+ 962us) server_negotiation.cc:407] Connection header received
0925 07:23:04.533139 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:04.533147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:04.533233 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:04.533395 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:04.534698 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.536203 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.537826 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.543464 (+ 5638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.544217 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.546326 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:04.546332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:04.546332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:04.546348 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:04.551789 (+ 5441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.552398 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.552403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.552403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.552510 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.552803 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.552803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.552804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.553059 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:04.553148 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:04.553481 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 07:23:04.553604 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260925 07:23:04.554615 17238 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:04.554742 17238 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.554931 17238 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
W20260925 07:23:04.555867 17238 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:39617 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:23:04.555953 17238 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.556022 17238 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:04.556335 17238 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.556414 17238 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
W20260925 07:23:04.556711 17238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:23:04.653270 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:04.653492 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:04.653518 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:04.654976 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:04.655040 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:04.655112 17242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:04.657374 17242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:37965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=41413
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:04.657678 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:04.657948 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:04.664752 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.664953 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.664884 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:04.665313 17242 server_base.cc:1034] running on GCE node
I20260925 07:23:04.665551 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:04.665845 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:04.667202 17242 hybrid_clock.cc:648] HybridClock initialized: now 1790320984667176 us; error 45 us; skew 500 ppm
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:04.670109 17242 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:04.670586 17242 webserver.cc:460] Webserver started at http://127.4.47.2:41413/ using document root <none> and password file <none>
I20260925 07:23:04.670773 17242 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:04.670818 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:04.690205 17242 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 07:23:04.691320 17258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.691494 17242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:04.691586 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "db9e4875ebee4ae68a4acdd9bd65ad38"
format_stamp: "Formatted at 2026-09-25 07:23:03 on dist-test-slave-zw07"
server_key: "b4fab308f89a1d4418345330f01e5f19"
server_key_iv: "37994f838e6380d5a08c365299607ae8"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.692095 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:04.742242 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:04.744818 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:04.744988 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:04.745270 17242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:04.745677 17242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:04.745726 17242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.745759 17242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:04.745782 17242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.752694 17242 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:37965
I20260925 07:23:04.752765 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:37965 every 8 connection(s)
I20260925 07:23:04.753142 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:23:04.755860 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17242
I20260925 07:23:04.756125 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.47.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:33761
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:39617
--webserver_port=42325
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:04.764842 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.753923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:37335 (local address 127.4.47.62:39617)
0925 07:23:04.753977 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:04.753979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:04.754218 (+ 239us) server_negotiation.cc:407] Connection header received
0925 07:23:04.754698 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:04.754700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:04.754716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:04.754803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:04.755415 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.756154 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.757741 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.758064 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.758108 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.761056 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:04.761058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:04.761058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:04.761066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:04.762940 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.763532 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.763533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.763534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.763595 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.763926 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.763927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.763928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.764080 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:04.764138 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:04.764552 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 07:23:04.764672 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:23:04.765522 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.753915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.2:37335)
0925 07:23:04.754071 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:04.754114 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:04.754590 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:04.754854 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:04.754857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:04.755024 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:04.755326 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.755333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.756268 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.756269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.757626 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.757629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.758320 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.758322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.758464 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.759095 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:04.759112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:04.760936 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:04.763044 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.763052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.763055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.763428 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.763691 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.763692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.763692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.763844 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.764177 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:04.764183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:04.764377 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 07:23:04.765026 (+ 649us) client_negotiation.cc:240] Negotiation successful
0925 07:23:04.765221 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 07:23:04.766078 17372 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:04.766192 17372 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.766402 17372 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
W20260925 07:23:04.767081 17372 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:39617 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:23:04.767180 17372 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.767257 17372 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:04.767566 17372 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.767663 17372 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
W20260925 07:23:04.767962 17372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:23:04.858723 17375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:04.858948 17375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:04.858989 17375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:04.860446 17375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:04.860523 17375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:04.860570 17375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:04.862622 17375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.47.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:33761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=42325
--tserver_master_addrs=127.4.47.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:04.862900 17375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:04.863216 17375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:04.867962 17380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.868340 17381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:04.868487 17383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:04.869486 17375 server_base.cc:1034] running on GCE node
I20260925 07:23:04.869709 17375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:04.869995 17375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:04.871136 17375 hybrid_clock.cc:648] HybridClock initialized: now 1790320984871120 us; error 25 us; skew 500 ppm
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:04.873984 17375 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:04.874508 17375 webserver.cc:460] Webserver started at http://127.4.47.3:42325/ using document root <none> and password file <none>
I20260925 07:23:04.874768 17375 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:04.874837 17375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:04.888864 17375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:04.889593 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.889802 17375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:04.889920 17375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "9a16cef8ed144575bc83fd398f7245e6"
format_stamp: "Formatted at 2026-09-25 07:23:04 on dist-test-slave-zw07"
server_key: "ccf8de1e2f7f8fb1807f86c474de4ca5"
server_key_iv: "91eeb4c3cbd169646b1a268e635e5364"
server_key_version: "kuduclusterkey@0"
I20260925 07:23:04.891208 17375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:04.938581 17375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:04.940836 17375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:04.940991 17375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:04.941213 17375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:04.941562 17375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:04.941607 17375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.941638 17375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:04.941656 17375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:04.948793 17375 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:33761
I20260925 07:23:04.948990 17505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:33761 every 8 connection(s)
I20260925 07:23:04.949244 17375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:04.953847 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17375
Sep 25 07:23:04 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:04.960314 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.950085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:39617 (local address 127.4.47.3:60753)
0925 07:23:04.950249 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:04.950301 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:04.950904 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:04.951189 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:04.951193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:04.951405 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:04.951606 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.951610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.952448 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.952449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.953461 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:04.953464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.953892 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.953893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:04.954044 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.954727 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:04.954738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:04.956444 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:04.958478 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.958486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.958489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.958822 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.959047 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:04.959048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:04.959049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:04.959230 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:04.959548 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:04.959555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:04.959756 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 07:23:04.959893 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 07:23:04.960063 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 07:23:04.960688 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.950110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:60753 (local address 127.4.47.62:39617)
0925 07:23:04.950219 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:04.950222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:04.950385 (+ 163us) server_negotiation.cc:407] Connection header received
0925 07:23:04.951016 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:04.951018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:04.951032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:04.951108 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:04.951716 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.952347 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.953576 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.953782 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.953866 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.956568 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:04.956570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:04.956570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:04.956578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:04.958382 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.958914 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.958915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.958916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.958976 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.959314 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.959315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.959315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.959454 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:04.959519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:04.959933 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 07:23:04.960297 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 07:23:04.961270 17506 heartbeater.cc:347] Connected to a master server at 127.4.47.62:39617
I20260925 07:23:04.961383 17506 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.961580 17506 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:04.962275 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:04.954324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.4.47.62:39617)
0925 07:23:04.954501 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:04.954504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:04.954513 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:04.954601 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:04.954602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:04.954616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:04.954698 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:04.955777 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.956342 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.957281 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:04.957536 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:04.957600 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:04.959193 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:04.959194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:04.959195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:04.959202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:04.960953 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.961317 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.961318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.961319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.961380 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:04.961596 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:04.961597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:04.961598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:04.961728 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:04.961917 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:04.962056 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 07:23:04.962160 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
W20260925 07:23:04.962298 17506 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:39617 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:23:04.962368 17506 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.962426 17506 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:04.962806 17506 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:04.962885 17506 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
W20260925 07:23:04.963201 17506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 07:23:05 dist-test-slave-zw07 krb5kdc[16043](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790320985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:05.556907 17238 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:05.557094 17238 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:05.557515 17040 ts_manager.cc:194] Registered new tserver with Master: 1dc05234cca64039af5502508a168a6f (127.4.47.1:33539)
I20260925 07:23:05.558524 17040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:59965
I20260925 07:23:05.768170 17372 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:05.768329 17372 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:05.768771 17040 ts_manager.cc:194] Registered new tserver with Master: db9e4875ebee4ae68a4acdd9bd65ad38 (127.4.47.2:37965)
I20260925 07:23:05.769328 17040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:37335
Sep 25 07:23:05 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:05.963400 17506 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:05.963572 17506 heartbeater.cc:511] Master 127.4.47.62:39617 requested a full tablet report, sending...
I20260925 07:23:05.963982 17040 ts_manager.cc:194] Registered new tserver with Master: 9a16cef8ed144575bc83fd398f7245e6 (127.4.47.3:33761)
I20260925 07:23:05.964499 17040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:60753
I20260925 07:23:05.966116 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:05.981622 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.967105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.4.47.62:39617)
0925 07:23:05.967232 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:05.967235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:05.967310 (+ 75us) server_negotiation.cc:407] Connection header received
0925 07:23:05.967381 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:05.967382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:05.967395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:05.967473 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:05.968170 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.968952 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.970016 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.970332 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.970410 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:05.978351 (+ 7941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:05.978354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:05.978354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:05.978363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:05.980261 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:05.980682 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:05.980684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:05.980684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:05.980752 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:05.980986 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:05.980987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:05.980988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:05.981186 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:05.981286 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:05.981426 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 07:23:05.981521 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260925 07:23:05.984234 17040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:23:05.986541 17040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:23:05.994778 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.989680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:37965 (local address 127.0.0.1:43288)
0925 07:23:05.990534 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:05.990576 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:05.990678 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:05.990898 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:05.990902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:05.990904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:05.990977 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.990979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.992282 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.992284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:05.993471 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.993474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.994481 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.994482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:05.994561 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:05.994573 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:23:05.994636 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 07:23:05.994686 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":107,"threads_started":1}
I20260925 07:23:05.995411 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.990033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.4.47.2:37965)
0925 07:23:05.990462 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:05.990468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:05.990640 (+ 172us) server_negotiation.cc:407] Connection header received
0925 07:23:05.990776 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:05.990779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:05.990797 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:05.990881 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:05.991048 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.992152 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.993577 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.994324 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.994392 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:05.994446 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:05.995287 (+ 841us) server_negotiation.cc:299] Negotiation successful
0925 07:23:05.995333 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":92,"threads_started":1}
I20260925 07:23:05.996284 17288 tablet_service.cc:1467] Processing CreateTablet for tablet f0425dcdd9524077997263353731af05 (DEFAULT_TABLE table=test-table [id=38df4a0349d6431c8309350cd7d86fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:05.996428 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.989681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:33539 (local address 127.0.0.1:60784)
0925 07:23:05.991138 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:05.991147 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:05.991189 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:05.991442 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:05.991443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:05.991444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:05.991480 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.991482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.993074 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.993075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:05.994206 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.994209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.996078 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.996079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:05.996216 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:05.996221 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:23:05.996301 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 07:23:05.996346 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260925 07:23:05.996665 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0425dcdd9524077997263353731af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:05.999541 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.990214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.4.47.1:33539)
0925 07:23:05.990347 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:05.990351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:05.991245 (+ 894us) server_negotiation.cc:407] Connection header received
0925 07:23:05.991283 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:05.991285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:05.991300 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:05.991366 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:05.991546 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.992988 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.994429 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.995919 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.995989 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:05.996044 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:05.999394 (+ 3350us) server_negotiation.cc:299] Negotiation successful
0925 07:23:05.999447 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 07:23:06.001202 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.990030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.4.47.3:33761)
0925 07:23:05.990184 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:05.990189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:05.994389 (+ 4200us) server_negotiation.cc:407] Connection header received
0925 07:23:05.994970 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:05.994973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:05.994991 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:05.995043 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:05.996541 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.998007 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.999708 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.000470 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.000518 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.000568 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:06.001071 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 07:23:06.001122 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260925 07:23:06.001200 17155 tablet_service.cc:1467] Processing CreateTablet for tablet f0425dcdd9524077997263353731af05 (DEFAULT_TABLE table=test-table [id=38df4a0349d6431c8309350cd7d86fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:06.001410 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:05.989936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:33761 (local address 127.0.0.1:35242)
0925 07:23:05.994324 (+ 4388us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:05.994335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:05.994872 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:05.995178 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:05.995179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:05.995181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:05.995220 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.995221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:05.998096 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:05.998097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:05.999283 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:05.999285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.000632 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.000633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:06.000697 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.000701 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:06.001281 (+ 580us) client_negotiation.cc:240] Negotiation successful
0925 07:23:06.001334 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4373,"thread_start_us":70,"threads_started":1}
I20260925 07:23:06.001607 17155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0425dcdd9524077997263353731af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:06.004596 17440 tablet_service.cc:1467] Processing CreateTablet for tablet f0425dcdd9524077997263353731af05 (DEFAULT_TABLE table=test-table [id=38df4a0349d6431c8309350cd7d86fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:06.004982 17440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0425dcdd9524077997263353731af05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:06.007489 17539 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: Bootstrap starting.
I20260925 07:23:06.008036 17540 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Bootstrap starting.
I20260925 07:23:06.008656 17540 tablet_bootstrap.cc:654] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:06.008967 17540 log.cc:824] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:06.009868 17540 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: No bootstrap required, opened a new log
I20260925 07:23:06.009949 17540 ts_tablet_manager.cc:1397] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:23:06.011252 17541 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Bootstrap starting.
I20260925 07:23:06.012012 17541 tablet_bootstrap.cc:654] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:06.012405 17541 log.cc:824] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:06.012681 17540 raft_consensus.cc:348] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.012873 17540 raft_consensus.cc:374] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:06.012955 17540 raft_consensus.cc:729] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc05234cca64039af5502508a168a6f, State: Initialized, Role: FOLLOWER
I20260925 07:23:06.013125 17540 consensus_queue.cc:260] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.013435 17540 ts_tablet_manager.cc:1428] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:23:06.013460 17238 heartbeater.cc:503] Master 127.4.47.62:39617 was elected leader, sending a full tablet report...
I20260925 07:23:06.015503 17539 tablet_bootstrap.cc:654] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:06.015893 17539 log.cc:824] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:06.016959 17539 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: No bootstrap required, opened a new log
I20260925 07:23:06.017053 17539 ts_tablet_manager.cc:1397] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 07:23:06.019593 17541 tablet_bootstrap.cc:492] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: No bootstrap required, opened a new log
I20260925 07:23:06.019691 17541 ts_tablet_manager.cc:1397] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 07:23:06.019925 17539 raft_consensus.cc:348] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.020210 17539 raft_consensus.cc:374] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:06.020255 17539 raft_consensus.cc:729] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9e4875ebee4ae68a4acdd9bd65ad38, State: Initialized, Role: FOLLOWER
I20260925 07:23:06.020416 17539 consensus_queue.cc:260] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.020721 17539 ts_tablet_manager.cc:1428] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 07:23:06.020848 17372 heartbeater.cc:503] Master 127.4.47.62:39617 was elected leader, sending a full tablet report...
I20260925 07:23:06.022300 17541 raft_consensus.cc:348] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.022472 17541 raft_consensus.cc:374] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:06.022500 17541 raft_consensus.cc:729] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a16cef8ed144575bc83fd398f7245e6, State: Initialized, Role: FOLLOWER
I20260925 07:23:06.022632 17541 consensus_queue.cc:260] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.022918 17541 ts_tablet_manager.cc:1428] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:23:06.022972 17506 heartbeater.cc:503] Master 127.4.47.62:39617 was elected leader, sending a full tablet report...
W20260925 07:23:06.030728 17239 tablet.cc:2369] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 07:23:06 dist-test-slave-zw07 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for HTTP/127.4.47.20@KRBTEST.COM
I20260925 07:23:06.159902 17546 raft_consensus.cc:482] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:06.160060 17546 raft_consensus.cc:504] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.160471 17546 leader_election.cc:288] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc05234cca64039af5502508a168a6f (127.4.47.1:33539), 9a16cef8ed144575bc83fd398f7245e6 (127.4.47.3:33761)
I20260925 07:23:06.169595 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.161241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:34817 (local address 127.4.47.1:33539)
0925 07:23:06.161303 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:06.161306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:06.162427 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 07:23:06.162471 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:06.162473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:06.162496 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:06.162572 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:06.163339 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.164048 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.168588 (+ 4540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.169182 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.169232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.169243 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:06.169480 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 07:23:06.169520 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:23:06.169929 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.160811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:33539 (local address 127.4.47.2:34817)
0925 07:23:06.162329 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:06.162343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:06.162419 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:06.163141 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:06.163142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:06.163145 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:06.163238 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:06.163241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.164155 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.164156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:06.165407 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:06.165410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.169331 (+ 3921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.169332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:06.169404 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.169408 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:06.169780 (+ 372us) client_negotiation.cc:240] Negotiation successful
0925 07:23:06.169834 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"thread_start_us":110,"threads_started":1}
I20260925 07:23:06.170819 17174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0425dcdd9524077997263353731af05" candidate_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc05234cca64039af5502508a168a6f" is_pre_election: true
I20260925 07:23:06.171005 17174 raft_consensus.cc:2454] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db9e4875ebee4ae68a4acdd9bd65ad38 in term 0.
I20260925 07:23:06.171267 17262 leader_election.cc:302] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc05234cca64039af5502508a168a6f, db9e4875ebee4ae68a4acdd9bd65ad38; no voters:
I20260925 07:23:06.171437 17546 raft_consensus.cc:2781] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:06.171494 17546 raft_consensus.cc:482] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:06.171525 17546 raft_consensus.cc:3037] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:06.172565 17546 raft_consensus.cc:504] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.172713 17546 leader_election.cc:288] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [CANDIDATE]: Term 1 election: Requested vote from peers 1dc05234cca64039af5502508a168a6f (127.4.47.1:33539), 9a16cef8ed144575bc83fd398f7245e6 (127.4.47.3:33761)
I20260925 07:23:06.173094 17174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0425dcdd9524077997263353731af05" candidate_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc05234cca64039af5502508a168a6f"
I20260925 07:23:06.173187 17174 raft_consensus.cc:3037] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:06.174037 17174 raft_consensus.cc:2454] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db9e4875ebee4ae68a4acdd9bd65ad38 in term 1.
I20260925 07:23:06.174211 17262 leader_election.cc:302] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc05234cca64039af5502508a168a6f, db9e4875ebee4ae68a4acdd9bd65ad38; no voters:
I20260925 07:23:06.174292 17546 raft_consensus.cc:2781] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:06.174453 17546 raft_consensus.cc:686] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 1 LEADER]: Becoming Leader. State: Replica: db9e4875ebee4ae68a4acdd9bd65ad38, State: Running, Role: LEADER
I20260925 07:23:06.174544 17546 consensus_queue.cc:237] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } }
I20260925 07:23:06.175565 17040 catalog_manager.cc:5495] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 reported cstate change: term changed from 0 to 1, leader changed from <none> to db9e4875ebee4ae68a4acdd9bd65ad38 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 } health_report { overall_health: UNKNOWN } } }
I20260925 07:23:06.175741 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.161299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:40363 (local address 127.4.47.3:33761)
0925 07:23:06.161352 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:06.161354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:06.169254 (+ 7900us) server_negotiation.cc:407] Connection header received
0925 07:23:06.169384 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:06.169386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:06.169400 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:06.169433 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:06.170010 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.170658 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.174101 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.174648 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.174715 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.174725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:06.175615 (+ 890us) server_negotiation.cc:299] Negotiation successful
0925 07:23:06.175668 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:23:06.176129 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.161051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:33761 (local address 127.4.47.2:40363)
0925 07:23:06.169182 (+ 8131us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:06.169196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:06.169289 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:06.169854 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:06.169855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:06.169858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:06.169921 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:06.169922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.171341 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.171342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:06.173997 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:06.174000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.175433 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.175435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:06.175521 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.175526 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:23:06.175865 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 07:23:06.175909 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8104,"thread_start_us":51,"threads_started":1}
I20260925 07:23:06.176443 17460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0425dcdd9524077997263353731af05" candidate_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a16cef8ed144575bc83fd398f7245e6" is_pre_election: true
I20260925 07:23:06.176597 17460 raft_consensus.cc:2454] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db9e4875ebee4ae68a4acdd9bd65ad38 in term 0.
I20260925 07:23:06.176709 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0425dcdd9524077997263353731af05" candidate_uuid: "db9e4875ebee4ae68a4acdd9bd65ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a16cef8ed144575bc83fd398f7245e6"
I20260925 07:23:06.176760 17459 raft_consensus.cc:3037] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:06.177539 17459 raft_consensus.cc:2454] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db9e4875ebee4ae68a4acdd9bd65ad38 in term 1.
I20260925 07:23:06.190371 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.186854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.4.47.2:37965)
0925 07:23:06.186907 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:06.186909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:06.186921 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:06.186975 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:06.186976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:06.187004 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:06.187162 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:06.187386 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.188120 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.189200 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:06.189488 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:06.189555 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:06.189770 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:06.189911 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:06.189992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:06.190253 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 07:23:06.190274 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:23:06.199486 17459 raft_consensus.cc:1261] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 1 FOLLOWER]: Refusing update from remote peer db9e4875ebee4ae68a4acdd9bd65ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:06.199851 17174 raft_consensus.cc:1261] T f0425dcdd9524077997263353731af05 P 1dc05234cca64039af5502508a168a6f [term 1 FOLLOWER]: Refusing update from remote peer db9e4875ebee4ae68a4acdd9bd65ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:06.200238 17546 consensus_queue.cc:1035] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a16cef8ed144575bc83fd398f7245e6" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 07:23:06.200343 17507 tablet.cc:2369] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:06.200433 17546 consensus_queue.cc:1035] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc05234cca64039af5502508a168a6f" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:06.208490 17558 mvcc.cc:205] Tried to move back new op lower bound from 7333154759466262528 to 7333154759371407360. Current Snapshot: MvccSnapshot[applied={T|T < 7333154759466262528 or (T in {7333154759466262528})}]
I20260925 07:23:06.208647 17040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:06.208743 17040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:06.209080 17556 mvcc.cc:205] Tried to move back new op lower bound from 7333154759466262528 to 7333154759371407360. Current Snapshot: MvccSnapshot[applied={T|T < 7333154759466262528 or (T in {7333154759466262528})}]
I20260925 07:23:06.209465 17040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6b07ab5444b4066b8c76e2fa29e2434: Sending DeleteTablet for 3 replicas of tablet f0425dcdd9524077997263353731af05
I20260925 07:23:06.209760 17288 tablet_service.cc:1514] Processing DeleteTablet for tablet f0425dcdd9524077997263353731af05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:06 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:43288
I20260925 07:23:06.209826 17155 tablet_service.cc:1514] Processing DeleteTablet for tablet f0425dcdd9524077997263353731af05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:06 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260925 07:23:06.210275 17440 tablet_service.cc:1514] Processing DeleteTablet for tablet f0425dcdd9524077997263353731af05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:06 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:35242
I20260925 07:23:06.210522 17566 tablet_replica.cc:330] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: stopping tablet replica
I20260925 07:23:06.210665 17566 raft_consensus.cc:2229] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:06.210670 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17078
I20260925 07:23:06.210846 17566 raft_consensus.cc:2258] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:06.211468 17566 ts_tablet_manager.cc:1905] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:06.213253 17566 ts_tablet_manager.cc:1918] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:06.213394 17566 log.cc:1196] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/f0425dcdd9524077997263353731af05
I20260925 07:23:06.213738 17566 ts_tablet_manager.cc:1939] T f0425dcdd9524077997263353731af05 P 9a16cef8ed144575bc83fd398f7245e6: Deleting consensus metadata
I20260925 07:23:06.214908 17027 catalog_manager.cc:4841] TS 9a16cef8ed144575bc83fd398f7245e6 (127.4.47.3:33761): tablet f0425dcdd9524077997263353731af05 (table test-table [id=38df4a0349d6431c8309350cd7d86fb3]) successfully deleted
I20260925 07:23:06.209368 17555 mvcc.cc:205] Tried to move back new op lower bound from 7333154759466262528 to 7333154759371407360. Current Snapshot: MvccSnapshot[applied={T|T < 7333154759466262528 or (T in {7333154759466262528})}]
W20260925 07:23:06.220474 17027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:33539 (error 108)
I20260925 07:23:06.221097 17564 tablet_replica.cc:330] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38: stopping tablet replica
I20260925 07:23:06.221225 17564 raft_consensus.cc:2229] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:23:06.221347 17564 raft_consensus.cc:2258] T f0425dcdd9524077997263353731af05 P db9e4875ebee4ae68a4acdd9bd65ad38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:06.221556 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17242
I20260925 07:23:06.230058 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.229914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:33539 (local address 127.0.0.1:60794)
0925 07:23:06.229975 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:06.230011 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:23:06.230110 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17375
W20260925 07:23:06.230229 17027 catalog_manager.cc:4584] TS 1dc05234cca64039af5502508a168a6f (127.4.47.1:33539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0425dcdd9524077997263353731af05: Network error: Client connection negotiation failed: client connection to 127.4.47.1:33539: connect: Connection refused (error 111)
I20260925 07:23:06.231354 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:06.231246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:37965 (local address 127.0.0.1:43314)
0925 07:23:06.231295 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:06.231317 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:37965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:23:06.240363 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17007
I20260925 07:23:06.246940 4284 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 07:23:06.591645 4284 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 07:23:06.591748 4284 mini_ranger.cc:66] Stopping Ranger...
I20260925 07:23:06.936503 4284 mini_ranger.cc:68] Stopped Ranger
2026-09-25 07:23:06.936 UTC [16073] LOG: received smart shutdown request
2026-09-25 07:23:06.941 UTC [16073] LOG: background worker "logical replication launcher" (PID 16082) exited with exit code 1
2026-09-25 07:23:06.941 UTC [16077] LOG: shutting down
2026-09-25 07:23:07.131 UTC [16073] LOG: database system is shut down
2026-09-25T07:23:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47295 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:23:07 dist-test-slave-zw07 krb5kdc[17576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:23:07 dist-test-slave-zw07 krb5kdc[17576](info): set up 2 sockets
Sep 25 07:23:07 dist-test-slave-zw07 krb5kdc[17576](info): commencing operation
krb5kdc: starting...
W20260925 07:23:08.283054 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:23:08.401901 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37625
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:08.482592 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:08.482808 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:08.482831 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:08.483930 17589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:08.483995 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:08.484011 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:08.484023 17589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:08.484035 17589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:08.485512 17589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:08.485708 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:08.485875 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:08.490671 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:08.490815 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:08.490931 17589 server_base.cc:1034] running on GCE node
W20260925 07:23:08.490846 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:08.491263 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:08.491560 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:08.492733 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790320988492719 us; error 57 us; skew 500 ppm
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:08.495225 17589 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:08.495695 17589 webserver.cc:460] Webserver started at http://127.4.47.62:37761/ using document root <none> and password file <none>
I20260925 07:23:08.495901 17589 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:08.495945 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:08.496037 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:08.497098 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "c06785073ece417eacf5f0fc84166ea4"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "6be93d643d7606e467c3a2ef1f1c55af"
server_key_iv: "b7ee4c3c88035e157aa4db5334c173dc"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.497411 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "c06785073ece417eacf5f0fc84166ea4"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "6be93d643d7606e467c3a2ef1f1c55af"
server_key_iv: "b7ee4c3c88035e157aa4db5334c173dc"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.498769 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:23:08.499599 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:08.499871 17589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:23:08.499950 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c06785073ece417eacf5f0fc84166ea4"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "6be93d643d7606e467c3a2ef1f1c55af"
server_key_iv: "b7ee4c3c88035e157aa4db5334c173dc"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.500043 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:08.519017 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:08.521320 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:08.521451 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:08.525856 17589 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:37625
I20260925 07:23:08.525885 17656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:37625 every 8 connection(s)
I20260925 07:23:08.526283 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:08.526952 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17589
I20260925 07:23:08.527040 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:23:08.527285 4284 external_mini_cluster.cc:1428] Setting key 41c3174e175c2cce4de988c535367f85
I20260925 07:23:08.527303 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:08.528913 17657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.47.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.61@KRBTEST.COM" created.
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:08.542804 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.529615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37625 (local address 127.0.0.1:56542)
0925 07:23:08.529830 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.529873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.530573 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.530925 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.530928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.531354 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.531589 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.531596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.532602 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.532603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.533781 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.533784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.534283 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.534284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.534377 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.534926 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.534939 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.539164 (+ 4225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.541314 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.541320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.541322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.541517 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.541763 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.541764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.541764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.541844 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.542352 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.542358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.542418 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.542497 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.542632 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260925 07:23:08.542898 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.529714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.4.47.62:37625)
0925 07:23:08.529853 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.529862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.529973 (+ 111us) server_negotiation.cc:407] Connection header received
0925 07:23:08.530748 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.530750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.530785 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.530889 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.531689 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.532500 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.533876 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.534188 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.534268 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.539256 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.539263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.539264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.539271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.541226 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.541582 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.541585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.541586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.541693 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.541912 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.541913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.541914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.542262 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.542360 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.542558 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.542722 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 07:23:08.543809 17657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:23:08.544169 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.544048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52130)
0925 07:23:08.544098 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.544142 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:23:08.544334 17607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
I20260925 07:23:08.544760 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.544678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52134)
0925 07:23:08.544720 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.544744 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:23:08.544893 17657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34409: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
I20260925 07:23:08.554754 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:34409
--webserver_interface=127.4.47.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:08.594188 17657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } attempt: 1
I20260925 07:23:08.594799 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.594643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52142)
0925 07:23:08.594711 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.594763 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 07:23:08.595317 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.595205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52156)
0925 07:23:08.595253 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.595285 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:23:08.595434 17657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34409: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
W20260925 07:23:08.635459 17664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:08.635675 17664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:08.635699 17664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:08.636919 17664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:08.636966 17664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:08.636981 17664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:08.637025 17664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:08.637058 17664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:08.638568 17664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.47.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:08.638832 17664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:08.639093 17664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:08.643990 17670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:08.643983 17672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:08.643989 17669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:08.644301 17664 server_base.cc:1034] running on GCE node
I20260925 07:23:08.644553 17664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:08.644853 17664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:08.646008 17664 hybrid_clock.cc:648] HybridClock initialized: now 1790320988645995 us; error 28 us; skew 500 ppm
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:08.648612 17664 init.cc:377] Logged in from keytab as kudu/127.4.47.61@KRBTEST.COM (short username kudu)
I20260925 07:23:08.649113 17664 webserver.cc:460] Webserver started at http://127.4.47.61:37027/ using document root <none> and password file <none>
I20260925 07:23:08.649358 17664 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:08.649426 17664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:08.649578 17664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:08.650667 17664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data/instance:
uuid: "23b66b8f7dcc4a728a2d6d4e8206233f"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "017d58591232fa130db322843a6791fd"
server_key_iv: "29080a967b51aabe17ef1777555ced9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.651041 17664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal/instance:
uuid: "23b66b8f7dcc4a728a2d6d4e8206233f"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "017d58591232fa130db322843a6791fd"
server_key_iv: "29080a967b51aabe17ef1777555ced9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.652407 17664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:08.653133 17679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:08.653311 17664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:08.653410 17664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "23b66b8f7dcc4a728a2d6d4e8206233f"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "017d58591232fa130db322843a6791fd"
server_key_iv: "29080a967b51aabe17ef1777555ced9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.653543 17664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:08.663569 17657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } attempt: 2
I20260925 07:23:08.664101 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.663963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52170)
0925 07:23:08.664026 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.664069 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:23:08.664454 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.664374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52186)
0925 07:23:08.664415 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.664439 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:23:08.664564 17657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34409: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34409: connect: Connection refused (error 111)
I20260925 07:23:08.667253 17664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:08.669595 17664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:08.669714 17664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:08.673938 17664 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.61:34409
I20260925 07:23:08.673993 17731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.61:34409 every 8 connection(s)
I20260925 07:23:08.674362 17664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
I20260925 07:23:08.675320 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:08.676934 17732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:23:08.679690 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17664
I20260925 07:23:08.679800 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-1/wal/instance
I20260925 07:23:08.679996 4284 external_mini_cluster.cc:1428] Setting key 2b5772733818d039279908ae104dbbd7
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:08.686977 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.677665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.4.47.62:37625)
0925 07:23:08.677713 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.677715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.677923 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:23:08.678390 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.678391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.678400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.678461 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.679218 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.679901 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.680981 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.681324 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.681399 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.683810 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.683812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.683812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.683821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.685543 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.685944 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.685945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.685945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.686014 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.686250 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.686251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.686251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.686428 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.686507 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.686750 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.686855 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:23:08.687026 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.677603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37625 (local address 127.0.0.1:56546)
0925 07:23:08.677776 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.677837 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.678296 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.678533 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.678536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.678812 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.679024 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.679028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.680006 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.680007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.680880 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.680883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.681433 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.681434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.681614 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.682194 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.682206 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.683709 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.685623 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.685630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.685632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.685865 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.686086 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.686087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.686087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.686184 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.686499 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.686503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.686618 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.686700 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.686833 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260925 07:23:08.687793 17732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.47.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.60@KRBTEST.COM" created.
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:23:08.695600 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.688136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.4.47.61:34409)
0925 07:23:08.688303 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.688308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.688326 (+ 18us) server_negotiation.cc:407] Connection header received
0925 07:23:08.688344 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.688346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.688357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.688418 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.688691 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.689492 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.690134 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.690469 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.690524 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.692570 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.692580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.692580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.692588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.694265 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.694711 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.694713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.694714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.694763 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.694984 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.694984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.694985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.695204 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.695268 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.695456 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.695522 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 07:23:08.695745 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.688061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52190)
0925 07:23:08.688115 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.688126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.688164 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.688450 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.688451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.688576 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.688624 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.688625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.689563 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.689564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.690061 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.690063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.690535 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.690535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.690587 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.691004 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.691009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.692483 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.694360 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.694361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.694362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.694620 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.694832 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.694833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.694833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.694913 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.695277 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.695278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.695336 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.695435 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.695594 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":47}
I20260925 07:23:08.696547 17732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:23:08.696995 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.696850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35760)
0925 07:23:08.696903 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.696956 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:23:08.697149 17680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
I20260925 07:23:08.697739 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.697583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35764)
0925 07:23:08.697647 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.697679 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 07:23:08.697929 17732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:33557: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
I20260925 07:23:08.709681 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:33557
--webserver_interface=127.4.47.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:08.747234 17732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } attempt: 1
I20260925 07:23:08.747780 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.747640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35766)
0925 07:23:08.747707 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.747749 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 07:23:08.748147 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.748071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35778)
0925 07:23:08.748109 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.748132 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 07:23:08.748245 17732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:33557: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
I20260925 07:23:08.755713 17657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } attempt: 3
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:23:08.762943 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.756065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52204)
0925 07:23:08.756118 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.756130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.756177 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.756405 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.756406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.756617 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.756696 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.756698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.757631 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.757632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.758159 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.758160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.758549 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.758550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.758624 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.759154 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.759159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.760358 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.761895 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.761896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.761896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.762069 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.762285 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.762286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.762286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.762349 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.762602 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.762604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.762651 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.762737 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.762846 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:23:08.762943 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.756126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.4.47.61:34409)
0925 07:23:08.756189 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.756191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.756203 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:08.756270 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.756271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.756287 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.756395 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.756776 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.757548 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.758246 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.758475 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.758554 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.760451 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.760452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.760452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.760461 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.761798 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.762143 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.762144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.762145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.762212 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.762413 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.762414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.762414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.762528 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.762600 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.762761 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.762843 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:23:08.763341 17657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:23:08.763684 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.763587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35780)
0925 07:23:08.763635 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.763660 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:23:08.764146 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.764005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35796)
0925 07:23:08.764083 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.764116 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 07:23:08.764288 17657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:33557: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
W20260925 07:23:08.792488 17739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:08.792740 17739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:08.792778 17739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:08.793919 17739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:08.793989 17739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:08.794016 17739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:08.794049 17739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:08.794073 17739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:08.795428 17657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } attempt: 1
I20260925 07:23:08.795718 17739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37625,127.4.47.61:34409,127.4.47.60:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.47.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:08.795866 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.795713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35810)
0925 07:23:08.795776 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.795816 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 07:23:08.795994 17739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:08.796264 17739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:23:08.796267 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.796157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35812)
0925 07:23:08.796210 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.796237 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 07:23:08.796375 17657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:33557: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
W20260925 07:23:08.801227 17745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:08.801234 17747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:08.801267 17744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:08.801575 17739 server_base.cc:1034] running on GCE node
I20260925 07:23:08.801829 17739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:08.802111 17739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:08.803303 17739 hybrid_clock.cc:648] HybridClock initialized: now 1790320988803289 us; error 28 us; skew 500 ppm
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:08.805931 17739 init.cc:377] Logged in from keytab as kudu/127.4.47.60@KRBTEST.COM (short username kudu)
I20260925 07:23:08.806391 17739 webserver.cc:460] Webserver started at http://127.4.47.60:37285/ using document root <none> and password file <none>
I20260925 07:23:08.806619 17739 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:08.806687 17739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:08.806845 17739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:08.808085 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data/instance:
uuid: "b0d50947775e485b8979bdceef846787"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "07524a3fb7454ed63aac77be9c088cfc"
server_key_iv: "c5018a863ecfe9592e27ea1c99882140"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.808461 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal/instance:
uuid: "b0d50947775e485b8979bdceef846787"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "07524a3fb7454ed63aac77be9c088cfc"
server_key_iv: "c5018a863ecfe9592e27ea1c99882140"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.809829 17739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:08.810570 17754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:08.810730 17739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:23:08.810832 17739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "b0d50947775e485b8979bdceef846787"
format_stamp: "Formatted at 2026-09-25 07:23:08 on dist-test-slave-zw07"
server_key: "07524a3fb7454ed63aac77be9c088cfc"
server_key_iv: "c5018a863ecfe9592e27ea1c99882140"
server_key_version: "encryptionkey@0"
I20260925 07:23:08.810922 17739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:08.816567 17732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } attempt: 2
I20260925 07:23:08.817031 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.816883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35824)
0925 07:23:08.816964 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.817003 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 07:23:08.817380 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.817284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35830)
0925 07:23:08.817331 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.817355 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:23:08.817485 17732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:33557: Network error: Client connection negotiation failed: client connection to 127.4.47.60:33557: connect: Connection refused (error 111)
I20260925 07:23:08.837869 17739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:08.840000 17739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:08.840183 17739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:08.845079 17739 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.60:33557
I20260925 07:23:08.845183 17806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.60:33557 every 8 connection(s)
I20260925 07:23:08.845548 17739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
I20260925 07:23:08.846470 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:08.848155 17807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:08.854416 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17739
I20260925 07:23:08.854528 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790320837143632-4284-0/minicluster-data/master-2/wal/instance
I20260925 07:23:08.854732 4284 external_mini_cluster.cc:1428] Setting key 2d7860159d6f64fc10865d94b622a6d6
I20260925 07:23:08.857954 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.848987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.4.47.62:37625)
0925 07:23:08.849061 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.849064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.849247 (+ 183us) server_negotiation.cc:407] Connection header received
0925 07:23:08.849777 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.849778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.849787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.849857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.850524 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.851003 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.852029 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.852211 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.852266 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.854727 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.854729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.854729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.854736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.856433 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.856921 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.856922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.856923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.856977 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.857247 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.857247 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.857248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.857373 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.857428 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.857756 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.857856 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:23:08.858035 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.848911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37625 (local address 127.0.0.1:56558)
0925 07:23:08.849111 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.849155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.849689 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.849940 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.849943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.850212 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.850434 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.850438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.851135 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.851136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.851949 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.851951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.852288 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.852289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.852411 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.852969 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.852978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.854633 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.856501 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.856506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.856508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.856849 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.857046 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.857047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.857047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.857176 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.857437 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.857442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.857632 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.857715 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.857845 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260925 07:23:08.858603 17807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:23:08.864447 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.855203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.4.47.60:33557)
0925 07:23:08.855560 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.855565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.855580 (+ 15us) server_negotiation.cc:407] Connection header received
0925 07:23:08.855598 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.855599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.855610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.855688 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.856331 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.857488 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.858500 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.858884 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.858949 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.861387 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.861394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.861395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.861422 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.863133 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.863553 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.863555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.863556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.863609 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.863810 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.863811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.863811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.863946 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.864122 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.864171 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.864295 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":103,"threads_started":1}
I20260925 07:23:08.865495 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.858880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34409 (local address 127.0.0.1:52216)
0925 07:23:08.858924 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.858934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.858965 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.859152 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.859154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.859318 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.859376 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.859377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.860000 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.860001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.860485 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.860486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.860846 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.860847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.860924 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.861258 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.861263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.862387 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.864060 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.864061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.864061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.864332 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.864574 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.864575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.864575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.864653 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.865141 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.865142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.865188 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.865260 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.865381 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:23:08.865512 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.858940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.4.47.61:34409)
0925 07:23:08.858981 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.858983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.858991 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:23:08.859018 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.859018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.859035 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.859144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.859461 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.859921 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.860569 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.860770 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.860850 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.862474 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.862475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.862476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.862483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.863963 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.864415 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.864416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.864416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.864474 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.864921 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.864922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.864923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.865064 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.865133 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.865301 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.865399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:23:08.865978 17807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:23:08.871537 17657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } attempt: 2
I20260925 07:23:08.873878 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.865332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.4.47.62:37625)
0925 07:23:08.865373 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.865375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.865382 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:23:08.865404 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.865405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.865414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.865475 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.866173 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.866671 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.867570 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.867818 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.867881 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.870153 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.870155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.870155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.870162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.872229 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.872661 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.872662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.872662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.872724 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.872968 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.872970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.872970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.873120 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.873652 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.873693 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.873769 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:23:08.874580 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.866231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35856)
0925 07:23:08.866781 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.866791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.866835 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.866962 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.866963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.867133 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.867166 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.867168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.867760 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.867760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.868274 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.868276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.868649 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.868650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.868703 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.869072 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.869076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.870632 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.873525 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.873525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.873526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.873679 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.873893 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.873893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.873894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.873952 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.874204 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.874204 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.874249 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.874326 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.874424 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533}
I20260925 07:23:08.874600 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.866303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.4.47.60:33557)
0925 07:23:08.866354 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.866355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.866839 (+ 484us) server_negotiation.cc:407] Connection header received
0925 07:23:08.866888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.866889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.866897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.866940 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.867236 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.867687 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.868356 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.868566 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.868648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.870723 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.870725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.870725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.870732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.873430 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.873760 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.873761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.873761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.873817 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.874025 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.874026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.874026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.874125 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.874202 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.874361 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.874468 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":30}
I20260925 07:23:08.876405 17807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: Bootstrap starting.
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:23:08.877195 17807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:08.877553 17807 log.cc:824] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:08.878465 17807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: No bootstrap required, opened a new log
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:23:08.880336 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.871951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35870)
0925 07:23:08.871994 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.872006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.872053 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.872391 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.872392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.872570 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.872620 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.872622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.874040 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.874041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.874811 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.874814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.875430 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.875431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.875487 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.875924 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.875929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.877367 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.879249 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.879250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.879250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.879426 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.879645 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.879646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.879647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.879712 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.879994 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.879995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.880040 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.880124 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.880226 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 07:23:08.880357 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.872008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.4.47.60:33557)
0925 07:23:08.872215 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.872217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.872225 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:23:08.872247 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.872247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.872256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.872373 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.872733 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.873937 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.874906 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.875372 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.875431 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.877472 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.877473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.877474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.877481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.879165 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.879514 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.879515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.879516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.879571 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.879776 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.879777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.879777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.879916 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.879988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.880149 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.880227 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260925 07:23:08.880909 17807 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.881070 17807 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:08.881119 17807 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d50947775e485b8979bdceef846787, State: Initialized, Role: FOLLOWER
I20260925 07:23:08.881248 17807 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.881613 17807 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:08.881834 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:08.882000 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:23:08.882387 17657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: Bootstrap starting.
I20260925 07:23:08.883225 17657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:08.883546 17657 log.cc:824] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:08.884078 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.875213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.4.47.61:34409)
0925 07:23:08.875256 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.875257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.875266 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:08.875375 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.875376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.875388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.875446 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.876039 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.876673 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.877485 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.877766 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.877836 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.880704 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.880705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.880706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.880713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.882513 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.882927 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.882928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.882929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.882998 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.883244 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.883245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.883245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.883414 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.883604 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.883882 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.883971 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:23:08.884929 17657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: No bootstrap required, opened a new log
I20260925 07:23:08.885335 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 07:23:08.883711 17825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:23:08.886653 17825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:23:08.884493 17739 master_runner.cc:186] Error getting master registration for 127.4.47.62:37625: OK, instance_id { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" instance_seqno: 1790320988524634 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:23:08.886852 17739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:23:08.886963 17657 raft_consensus.cc:348] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.887115 17657 raft_consensus.cc:374] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:08.887158 17657 raft_consensus.cc:729] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06785073ece417eacf5f0fc84166ea4, State: Initialized, Role: FOLLOWER
I20260925 07:23:08.887290 17657 consensus_queue.cc:260] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.887643 17657 sys_catalog.cc:564] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:08.888173 17833 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:08.888446 17833 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:23:08.889489 17846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:23:08.889575 17846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:23:08.890273 17589 master_runner.cc:186] Error getting master registration for 127.4.47.61:34409: OK, instance_id { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" instance_seqno: 1790320988672713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:23:08.893518 17589 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:23:08.898824 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.886911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.4.47.62:37625)
0925 07:23:08.886950 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.886952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.887253 (+ 301us) server_negotiation.cc:407] Connection header received
0925 07:23:08.887391 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.887391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.887402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.887432 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.888402 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.889129 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.890474 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.890766 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.890850 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.894335 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.894336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.894337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.894344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.896557 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.896959 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.896960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.896961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.897018 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.897237 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.897237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.897238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.897384 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.897698 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.898411 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.898725 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:23:08.901364 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.887589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.4.47.61:34409)
0925 07:23:08.887655 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.887657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.890570 (+ 2913us) server_negotiation.cc:407] Connection header received
0925 07:23:08.890664 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.890664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.890674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.890703 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.891463 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.892766 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.894602 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.894914 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.894961 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.897245 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.897246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.897247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.897253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.899150 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.900451 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.900452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.900452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.900514 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.900717 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.900718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.900718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.900850 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.900940 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.901158 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.901243 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:23:08.901897 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.888076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.4.47.60:33557)
0925 07:23:08.888116 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.888118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.891727 (+ 3609us) server_negotiation.cc:407] Connection header received
0925 07:23:08.891833 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.891834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.891852 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.891886 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.892709 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.893446 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.894473 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.894809 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.894852 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.897533 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.897534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.897534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.897542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.900041 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.900770 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.900771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.900772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.900834 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.901046 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.901047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.901047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.901180 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.901259 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.901707 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.901794 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:23:08.908625 17732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } attempt: 3
Sep 25 07:23:08 dist-test-slave-zw07 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320988, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:23:08.915978 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.909072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.4.47.60:33557)
0925 07:23:08.909125 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:08.909126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:08.909133 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:23:08.909152 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:08.909153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:08.909169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:08.909236 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:08.909520 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.909995 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.910678 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.910877 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.910973 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.913180 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:08.913182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:08.913182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:08.913189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:08.914784 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.915184 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.915186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.915186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.915239 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:08.915462 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:08.915462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:08.915463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:08.915578 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:08.915653 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:08.915804 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 07:23:08.915881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:23:08.915979 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:08.908923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:33557 (local address 127.0.0.1:35894)
0925 07:23:08.908968 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:08.908979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:08.909018 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:08.909240 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:08.909241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:08.909399 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:08.909437 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.909438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.910076 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.910077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.910583 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:08.910584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:08.910970 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:08.910971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:08.911028 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:08.911522 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:08.911527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:08.913086 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:08.914896 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.914897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.914897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.915126 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.915332 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:08.915333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:08.915333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:08.915390 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:08.915645 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:08.915646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:08.915692 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 07:23:08.915779 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:23:08.915881 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:23:08.917593 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: Bootstrap starting.
I20260925 07:23:08.918257 17732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:08.918565 17732 log.cc:824] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:08.919510 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: No bootstrap required, opened a new log
I20260925 07:23:08.921635 17732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.921767 17732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:08.921820 17732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b66b8f7dcc4a728a2d6d4e8206233f, State: Initialized, Role: FOLLOWER
I20260925 07:23:08.921950 17732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:08.922227 17849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:08.922291 17732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:08.922367 17849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:23:08.924134 17860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:23:08.924234 17860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:23:08.926282 17664 master_runner.cc:263] No leader master found from master 23b66b8f7dcc4a728a2d6d4e8206233f
I20260925 07:23:08.926338 17664 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 07:23:09.022387 17610 tablet.cc:2369] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:23:09.090854 17760 tablet.cc:2369] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:23:09.170502 17685 tablet.cc:2369] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:09.175827 17849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:09.175957 17849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:09.176303 17849 leader_election.cc:288] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c06785073ece417eacf5f0fc84166ea4 (127.4.47.62:37625), b0d50947775e485b8979bdceef846787 (127.4.47.60:33557)
I20260925 07:23:09.176663 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d50947775e485b8979bdceef846787" is_pre_election: true
I20260925 07:23:09.176683 17631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06785073ece417eacf5f0fc84166ea4" is_pre_election: true
I20260925 07:23:09.176859 17781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b66b8f7dcc4a728a2d6d4e8206233f in term 0.
I20260925 07:23:09.176857 17631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23b66b8f7dcc4a728a2d6d4e8206233f in term 0.
I20260925 07:23:09.177081 17680 leader_election.cc:302] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b66b8f7dcc4a728a2d6d4e8206233f, b0d50947775e485b8979bdceef846787; no voters:
I20260925 07:23:09.177241 17849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:09.177338 17849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:09.177374 17849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:09.178099 17849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:09.178243 17849 leader_election.cc:288] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [CANDIDATE]: Term 1 election: Requested vote from peers c06785073ece417eacf5f0fc84166ea4 (127.4.47.62:37625), b0d50947775e485b8979bdceef846787 (127.4.47.60:33557)
I20260925 07:23:09.178452 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d50947775e485b8979bdceef846787"
I20260925 07:23:09.178472 17631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06785073ece417eacf5f0fc84166ea4"
I20260925 07:23:09.178575 17781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:09.178579 17631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:09.179479 17781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b66b8f7dcc4a728a2d6d4e8206233f in term 1.
I20260925 07:23:09.179479 17631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23b66b8f7dcc4a728a2d6d4e8206233f in term 1.
I20260925 07:23:09.179662 17680 leader_election.cc:302] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b66b8f7dcc4a728a2d6d4e8206233f, b0d50947775e485b8979bdceef846787; no voters:
I20260925 07:23:09.179766 17849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:09.179886 17849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [term 1 LEADER]: Becoming Leader. State: Replica: 23b66b8f7dcc4a728a2d6d4e8206233f, State: Running, Role: LEADER
I20260925 07:23:09.179978 17849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } }
I20260925 07:23:09.180369 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b66b8f7dcc4a728a2d6d4e8206233f. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.180471 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:09.180677 17864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:09.180869 17864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:09.182371 17631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [term 1 FOLLOWER]: Refusing update from remote peer 23b66b8f7dcc4a728a2d6d4e8206233f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:09.182408 17781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [term 1 FOLLOWER]: Refusing update from remote peer 23b66b8f7dcc4a728a2d6d4e8206233f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:09.182678 17849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:09.182780 17862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:09.183987 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b66b8f7dcc4a728a2d6d4e8206233f. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.184082 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:23:09.184053 17833 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b66b8f7dcc4a728a2d6d4e8206233f. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.184145 17833 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:23:09.185015 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.185109 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:09.185286 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.185302 17833 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.185380 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:23:09.185384 17833 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:23:09.185051 17870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06785073ece417eacf5f0fc84166ea4" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37625 } } peers { permanent_uuid: "23b66b8f7dcc4a728a2d6d4e8206233f" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34409 } } peers { permanent_uuid: "b0d50947775e485b8979bdceef846787" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 33557 } } }
I20260925 07:23:09.185500 17866 mvcc.cc:205] Tried to move back new op lower bound from 7333154771688284160 to 7333154771681824768. Current Snapshot: MvccSnapshot[applied={T|T < 7333154771688284160}]
I20260925 07:23:09.185580 17870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:09.185635 17867 mvcc.cc:205] Tried to move back new op lower bound from 7333154771688284160 to 7333154771681824768. Current Snapshot: MvccSnapshot[applied={T|T < 7333154771688284160}]
I20260925 07:23:09.185846 17864 catalog_manager.cc:1347] Generated new cluster ID: 89f26d7afcb84b93a2e3a4560d4648d1
I20260925 07:23:09.185894 17864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:23:09.194432 17864 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:23:09.195179 17864 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:09.204254 17864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b66b8f7dcc4a728a2d6d4e8206233f: Generated new TSK 0
I20260925 07:23:09.204517 17864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:23:09.887243 17825 catalog_manager.cc:1259] Loaded cluster ID: 89f26d7afcb84b93a2e3a4560d4648d1
I20260925 07:23:09.887318 17825 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: loading cluster ID for follower catalog manager: success
I20260925 07:23:09.888293 17825 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: acquiring CA information for follower catalog manager: success
I20260925 07:23:09.888612 17825 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0d50947775e485b8979bdceef846787: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:23:09.889971 17846 catalog_manager.cc:1259] Loaded cluster ID: 89f26d7afcb84b93a2e3a4560d4648d1
I20260925 07:23:09.890035 17846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: loading cluster ID for follower catalog manager: success
I20260925 07:23:09.890774 17846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: acquiring CA information for follower catalog manager: success
I20260925 07:23:09.891021 17846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c06785073ece417eacf5f0fc84166ea4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:23:10.260134 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17589
I20260925 07:23:10.265426 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17664
I20260925 07:23:10.270231 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17739
2026-09-25T07:23:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3025 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:23:10 dist-test-slave-zw07 krb5kdc[17879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:23:10 dist-test-slave-zw07 krb5kdc[17879](info): set up 2 sockets
Sep 25 07:23:10 dist-test-slave-zw07 krb5kdc[17879](info): commencing operation
krb5kdc: starting...
W20260925 07:23:11.311246 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:23:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:23:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:23:11.433854 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34947
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:34947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:11.514925 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:11.515219 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:11.515247 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:11.516417 17892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:11.516458 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:11.516472 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:11.516513 17892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:11.516539 17892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:11.518108 17892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:34947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:11.518356 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:11.518600 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:11.523663 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:11.523754 17892 server_base.cc:1034] running on GCE node
W20260925 07:23:11.523671 17898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:11.523671 17900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:11.524174 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:11.524477 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:11.525650 17892 hybrid_clock.cc:648] HybridClock initialized: now 1790320991525638 us; error 45 us; skew 500 ppm
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:11.539431 17892 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:11.539974 17892 webserver.cc:460] Webserver started at http://127.4.47.62:44269/ using document root <none> and password file <none>
I20260925 07:23:11.540215 17892 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:11.540287 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:11.540488 17892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:11.541671 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "b078890f16f34111bb24bf21410239c9"
format_stamp: "Formatted at 2026-09-25 07:23:11 on dist-test-slave-zw07"
server_key: "2e7f7c4e7de7e5f17f2c843e84a9fd4e"
server_key_iv: "9e490da01c1ecb172ea204e5428236cd"
server_key_version: "encryptionkey@0"
I20260925 07:23:11.542088 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "b078890f16f34111bb24bf21410239c9"
format_stamp: "Formatted at 2026-09-25 07:23:11 on dist-test-slave-zw07"
server_key: "2e7f7c4e7de7e5f17f2c843e84a9fd4e"
server_key_iv: "9e490da01c1ecb172ea204e5428236cd"
server_key_version: "encryptionkey@0"
I20260925 07:23:11.543562 17892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:23:11.544313 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:11.544489 17892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:23:11.544591 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "b078890f16f34111bb24bf21410239c9"
format_stamp: "Formatted at 2026-09-25 07:23:11 on dist-test-slave-zw07"
server_key: "2e7f7c4e7de7e5f17f2c843e84a9fd4e"
server_key_iv: "9e490da01c1ecb172ea204e5428236cd"
server_key_version: "encryptionkey@0"
I20260925 07:23:11.544672 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:11.564941 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:11.566911 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:11.567176 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:11.571206 17892 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:34947
I20260925 07:23:11.571244 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:34947 every 8 connection(s)
I20260925 07:23:11.571612 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:11.572495 17960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:11.574800 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9: Bootstrap starting.
I20260925 07:23:11.575454 17960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:11.575752 17960 log.cc:824] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:11.576469 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9: No bootstrap required, opened a new log
I20260925 07:23:11.578052 17960 raft_consensus.cc:348] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } }
I20260925 07:23:11.578193 17960 raft_consensus.cc:374] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:11.578215 17960 raft_consensus.cc:729] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b078890f16f34111bb24bf21410239c9, State: Initialized, Role: FOLLOWER
I20260925 07:23:11.578388 17960 consensus_queue.cc:260] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } }
I20260925 07:23:11.578455 17960 raft_consensus.cc:388] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:11.578497 17960 raft_consensus.cc:482] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:11.578547 17960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:11.578918 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 17892
I20260925 07:23:11.579034 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:23:11.579190 17960 raft_consensus.cc:504] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } }
I20260925 07:23:11.579236 4284 external_mini_cluster.cc:1428] Setting key 0455566457cdcfdb5506ae14ae83d764
I20260925 07:23:11.579279 17960 leader_election.cc:302] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b078890f16f34111bb24bf21410239c9; no voters:
I20260925 07:23:11.579473 17960 leader_election.cc:288] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:23:11.579497 17963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:11.579617 17963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [term 1 LEADER]: Becoming Leader. State: Replica: b078890f16f34111bb24bf21410239c9, State: Running, Role: LEADER
I20260925 07:23:11.579739 17960 sys_catalog.cc:564] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:11.579769 17963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } }
I20260925 07:23:11.580157 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b078890f16f34111bb24bf21410239c9. Latest consensus state: current_term: 1 leader_uuid: "b078890f16f34111bb24bf21410239c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } } }
I20260925 07:23:11.580308 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:11.580333 17965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b078890f16f34111bb24bf21410239c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078890f16f34111bb24bf21410239c9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34947 } } }
I20260925 07:23:11.580482 17965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:11.580561 17974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:11.581153 17974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:11.583014 17974 catalog_manager.cc:1347] Generated new cluster ID: 3ccacad4f8264236a907a6d2ce98e9a8
I20260925 07:23:11.583081 17974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:11.592304 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:11.579904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.4.47.62:34947)
0925 07:23:11.580593 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:11.580602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:11.580636 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:23:11.580956 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:11.580964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:11.581024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:11.581719 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:11.581942 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:11.583237 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:11.584301 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:11.584918 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:11.585014 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:11.587891 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:11.587895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:11.587896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:11.587908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:11.590589 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:11.591001 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:11.591005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:11.591005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:11.591152 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:11.591367 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:11.591368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:11.591368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:11.591687 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:11.591842 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:11.591917 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:23:11.592060 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
I20260925 07:23:11.593389 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:23:11.594587 17974 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:23:11.595156 17974 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:11.605748 17974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b078890f16f34111bb24bf21410239c9: Generated new TSK 0
I20260925 07:23:11.605980 17974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:23:11.647920 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 17892
2026-09-25T07:23:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1375 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17987](info): set up 2 sockets
Sep 25 07:23:11 dist-test-slave-zw07 krb5kdc[17987](info): commencing operation
krb5kdc: starting...
W20260925 07:23:12.687299 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:23:12 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:23:12.810519 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:12.892338 18000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:12.892585 18000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:12.892621 18000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:12.893800 18000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:12.893867 18000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:12.893890 18000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:12.893909 18000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:12.893944 18000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:12.895407 18000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:12.895694 18000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:12.895921 18000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:12.901002 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:12.900997 18008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:12.901049 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:12.901324 18000 server_base.cc:1034] running on GCE node
I20260925 07:23:12.901573 18000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:12.901865 18000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:12.903040 18000 hybrid_clock.cc:648] HybridClock initialized: now 1790320992903004 us; error 47 us; skew 500 ppm
Sep 25 07:23:12 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320992, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:12.905674 18000 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:12.906119 18000 webserver.cc:460] Webserver started at http://127.4.47.62:36067/ using document root <none> and password file <none>
I20260925 07:23:12.906374 18000 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:12.906445 18000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:12.906592 18000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:12.907740 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "ce49a948b52f4233a05408d6dbe9e0ca"
format_stamp: "Formatted at 2026-09-25 07:23:12 on dist-test-slave-zw07"
server_key: "ad77ddb435029f7d24fee8e68e95cdb4"
server_key_iv: "531e57a8e5982c101a01eaddb479dc9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:12.908160 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "ce49a948b52f4233a05408d6dbe9e0ca"
format_stamp: "Formatted at 2026-09-25 07:23:12 on dist-test-slave-zw07"
server_key: "ad77ddb435029f7d24fee8e68e95cdb4"
server_key_iv: "531e57a8e5982c101a01eaddb479dc9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:12.909608 18000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:23:12.910461 18015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:12.910650 18000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:12.910759 18000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "ce49a948b52f4233a05408d6dbe9e0ca"
format_stamp: "Formatted at 2026-09-25 07:23:12 on dist-test-slave-zw07"
server_key: "ad77ddb435029f7d24fee8e68e95cdb4"
server_key_iv: "531e57a8e5982c101a01eaddb479dc9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:12.910858 18000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:12.930450 18000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:12.932655 18000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:12.932840 18000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:12.937211 18000 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33467
I20260925 07:23:12.937243 18067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33467 every 8 connection(s)
I20260925 07:23:12.937630 18000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:12.938558 18068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:12.940948 18068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap starting.
I20260925 07:23:12.941551 18068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:12.941851 18068 log.cc:824] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:12.942596 18068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: No bootstrap required, opened a new log
I20260925 07:23:12.944343 18068 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:12.944491 18068 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:12.944514 18068 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Initialized, Role: FOLLOWER
I20260925 07:23:12.944677 18068 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:12.944741 18068 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:12.944763 18068 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:12.944813 18068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:12.945674 18068 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:12.945719 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18000
I20260925 07:23:12.945762 18068 leader_election.cc:302] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce49a948b52f4233a05408d6dbe9e0ca; no voters:
I20260925 07:23:12.945895 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:23:12.945979 18068 leader_election.cc:288] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:23:12.946004 18071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:12.946115 18071 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 LEADER]: Becoming Leader. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Running, Role: LEADER
I20260925 07:23:12.946131 4284 external_mini_cluster.cc:1428] Setting key 875df79e1f28b5570ed4c2cca4bfe79e
I20260925 07:23:12.946257 18068 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:12.946255 18071 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:12.946943 18072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:12.947036 18072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:12.947213 18073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce49a948b52f4233a05408d6dbe9e0ca. Latest consensus state: current_term: 1 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:12.947261 18073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:12.947625 18082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:12.948258 18082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:12.949643 18082 catalog_manager.cc:1347] Generated new cluster ID: bc728b7f660844049a3c4452cb898d30
I20260925 07:23:12.949697 18082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:23:12 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:12.958724 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:12.946746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.4.47.62:33467)
0925 07:23:12.946921 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:12.946931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:12.946963 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:23:12.947107 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:12.947115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:12.947175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:12.948041 (+ 866us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:12.948269 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:12.949631 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:12.950745 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:12.951447 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:12.951561 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:12.954336 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:12.954339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:12.954340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:12.954353 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:12.957000 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:12.957426 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:12.957429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:12.957429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:12.957554 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:12.957770 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:12.957771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:12.957771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:12.958085 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:12.958219 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:12.958344 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 07:23:12.958475 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 07:23:12.963135 18082 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:23:12.963804 18082 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:12.970249 18082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Generated new TSK 0
I20260925 07:23:12.970425 18082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:23:12.998697 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:13.081396 18094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.081598 18094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.081621 18094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.082811 18094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.082856 18094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.082883 18094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:13.084354 18094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.084597 18094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.084854 18094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.089875 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.089911 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.089875 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.090065 18094 server_base.cc:1034] running on GCE node
I20260925 07:23:13.090261 18094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.090452 18094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.091588 18094 hybrid_clock.cc:648] HybridClock initialized: now 1790320993091577 us; error 27 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.094095 18094 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:13.094544 18094 webserver.cc:460] Webserver started at http://127.4.47.1:43885/ using document root <none> and password file <none>
I20260925 07:23:13.094743 18094 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.094782 18094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.094867 18094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:13.095916 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "61b5aab706444090a161b861078359cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "ae715abaec34100c3bc3870898731e9c"
server_key_iv: "6e563441feb613054e27f6e1151da776"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.096212 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "61b5aab706444090a161b861078359cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "ae715abaec34100c3bc3870898731e9c"
server_key_iv: "6e563441feb613054e27f6e1151da776"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.097570 18094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:13.098255 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.098446 18094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:23:13.098513 18094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "61b5aab706444090a161b861078359cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "ae715abaec34100c3bc3870898731e9c"
server_key_iv: "6e563441feb613054e27f6e1151da776"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.098565 18094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:13.113168 18094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:13.115459 18094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:13.115577 18094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:13.115805 18094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:13.116137 18094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:13.116171 18094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.116194 18094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:13.116228 18094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.122054 18094 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:32819
I20260925 07:23:13.122103 18222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:32819 every 8 connection(s)
I20260925 07:23:13.122499 18094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:13.123865 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18094
I20260925 07:23:13.123984 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:23:13.124182 4284 external_mini_cluster.cc:1428] Setting key 845b7090c61e3a2611e9ad22b25934b6
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
I20260925 07:23:13.133553 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.123360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:34369 (local address 127.4.47.62:33467)
0925 07:23:13.123435 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.123437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.123629 (+ 192us) server_negotiation.cc:407] Connection header received
0925 07:23:13.124409 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.124410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.124423 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.124467 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:13.125821 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.126317 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.127518 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.127846 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.127920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.130528 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:13.130529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:13.130530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:13.130537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:13.132050 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.132575 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.132576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.132576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.132631 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.132960 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.132961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.132961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.133070 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:13.133124 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.133413 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.133488 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:23:13.133764 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.123317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:34369)
0925 07:23:13.123500 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.123538 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.124306 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.124583 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.124589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.125415 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:13.125706 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.125713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.126461 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.126462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.127403 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.127406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.127937 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.127939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.128057 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.128587 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:13.128596 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:13.130433 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:13.132144 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.132152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.132155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.132484 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.132702 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.132703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.132703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.132854 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.133142 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:13.133150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:13.133294 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.133390 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.133512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260925 07:23:13.134330 18223 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:13.134438 18223 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:13.134625 18223 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:13.135223 18032 ts_manager.cc:194] Registered new tserver with Master: 61b5aab706444090a161b861078359cf (127.4.47.1:32819)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
I20260925 07:23:13.135960 18032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:34369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:23:13.150280 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:13.231158 18228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.231361 18228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.231384 18228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.232547 18228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.232589 18228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.232616 18228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:13.234066 18228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.234320 18228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.234571 18228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.239606 18233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.239674 18234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.239817 18228 server_base.cc:1034] running on GCE node
W20260925 07:23:13.239639 18236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.240111 18228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.240309 18228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.241484 18228 hybrid_clock.cc:648] HybridClock initialized: now 1790320993241479 us; error 37 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.243934 18228 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:13.244407 18228 webserver.cc:460] Webserver started at http://127.4.47.2:43189/ using document root <none> and password file <none>
I20260925 07:23:13.244644 18228 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.244711 18228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.244858 18228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:13.245896 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "f5864ecc46254d9f9c0a9407d053a62e"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "b11cb2a06a5eefb513e5a46044b85262"
server_key_iv: "0d9166ee5c0e086e5d78baaab57f25bf"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.246282 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "f5864ecc46254d9f9c0a9407d053a62e"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "b11cb2a06a5eefb513e5a46044b85262"
server_key_iv: "0d9166ee5c0e086e5d78baaab57f25bf"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.247785 18228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:23:13.248554 18243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.248755 18228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:13.248860 18228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "f5864ecc46254d9f9c0a9407d053a62e"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "b11cb2a06a5eefb513e5a46044b85262"
server_key_iv: "0d9166ee5c0e086e5d78baaab57f25bf"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.248934 18228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:13.259394 18228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:13.261286 18228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:13.261449 18228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:13.261672 18228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:13.261968 18228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:13.262023 18228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.262063 18228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:13.262091 18228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.268450 18228 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:45969
I20260925 07:23:13.268501 18356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:45969 every 8 connection(s)
I20260925 07:23:13.268841 18228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:13.275161 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18228
I20260925 07:23:13.275269 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:23:13.275493 4284 external_mini_cluster.cc:1428] Setting key 9b36988a4074c59f39cf8e4a6e927848
I20260925 07:23:13.278798 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.269590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:42687 (local address 127.4.47.62:33467)
0925 07:23:13.269634 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.269636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.269859 (+ 223us) server_negotiation.cc:407] Connection header received
0925 07:23:13.270416 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.270417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.270427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.270502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:13.271524 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.272041 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.273011 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.273196 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.273245 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.275477 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:13.275478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:13.275479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:13.275485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:13.277083 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.277653 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.277655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.277656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.277725 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.278017 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.278018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.278019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.278171 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:13.278246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.278561 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.278670 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:23:13.278863 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.269536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:42687)
0925 07:23:13.269704 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.269746 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.270300 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.270573 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.270576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.271237 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:13.271438 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.271443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.272121 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.272122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.272935 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.272937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.273265 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.273266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.273360 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.273834 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:13.273843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:13.275382 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:13.277174 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.277183 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.277185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.277542 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.277810 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.277811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.277812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.277950 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.278294 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:13.278299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:13.278429 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.278516 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.278648 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260925 07:23:13.279395 18357 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:13.279497 18357 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:13.279718 18357 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:13.280145 18032 ts_manager.cc:194] Registered new tserver with Master: f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:13.280562 18032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:42687
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:23:13.301733 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:13.382429 18362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.382688 18362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.382728 18362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.383924 18362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.383998 18362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.384043 18362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:13.385509 18362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.385778 18362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.386013 18362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.391000 18370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.390985 18368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.390977 18367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.391348 18362 server_base.cc:1034] running on GCE node
I20260925 07:23:13.391515 18362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.391695 18362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.392853 18362 hybrid_clock.cc:648] HybridClock initialized: now 1790320993392830 us; error 28 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.395298 18362 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:13.395740 18362 webserver.cc:460] Webserver started at http://127.4.47.3:39967/ using document root <none> and password file <none>
I20260925 07:23:13.395924 18362 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.395965 18362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.396061 18362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:13.397051 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "f694befd00ac4026968dc07322ee45cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "519720fc52187e4c52571093b01afe0e"
server_key_iv: "e9f4a974c465c054739525dcdd9a33ce"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.397351 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "f694befd00ac4026968dc07322ee45cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "519720fc52187e4c52571093b01afe0e"
server_key_iv: "e9f4a974c465c054739525dcdd9a33ce"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.398567 18362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:13.399260 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.399448 18362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:13.399554 18362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "f694befd00ac4026968dc07322ee45cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "519720fc52187e4c52571093b01afe0e"
server_key_iv: "e9f4a974c465c054739525dcdd9a33ce"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.399627 18362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:13.420169 18362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:13.422077 18362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:13.422261 18362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:13.422511 18362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:13.422824 18362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:13.422879 18362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.422920 18362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:13.422960 18362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.428869 18362 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:34589
I20260925 07:23:13.428918 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:34589 every 8 connection(s)
I20260925 07:23:13.429224 18362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:13.436591 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18362
I20260925 07:23:13.436710 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:23:13.436914 4284 external_mini_cluster.cc:1428] Setting key 7bbd0ad678325466787d3ab99a30d424
I20260925 07:23:13.438582 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.429940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:37195 (local address 127.4.47.62:33467)
0925 07:23:13.430014 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.430017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.430225 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:23:13.430693 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.430694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.430704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.430764 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:13.431755 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.432240 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.433180 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.433357 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.433417 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.435521 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:13.435522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:13.435522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:13.435529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:13.437030 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.437582 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.437584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.437584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.437647 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.437930 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.437931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.437932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.438051 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:13.438129 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.438407 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.438487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:23:13.438784 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.429909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:37195)
0925 07:23:13.430084 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.430136 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.430611 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.430816 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.430825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.431460 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:13.431673 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.431678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.432318 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.432319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.433106 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.433108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.433437 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.433438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.433537 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.434002 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:13.434010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:13.435395 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:13.437127 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.437133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.437135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.437491 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.437740 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.437741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.437742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.437859 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.438121 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:13.438127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:13.438273 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.438389 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.438552 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 07:23:13.439249 18491 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:13.439332 18491 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:13.439467 18491 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:13.439792 18032 ts_manager.cc:194] Registered new tserver with Master: f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589)
I20260925 07:23:13.440169 18032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:37195
I20260925 07:23:13.448647 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:13.455919 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.449996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.4.47.62:33467)
0925 07:23:13.450055 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.450057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.450185 (+ 128us) server_negotiation.cc:407] Connection header received
0925 07:23:13.450224 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.450225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.450233 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.450289 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:13.450808 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.451359 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.452162 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.452342 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.452405 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.453584 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:13.453585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:13.453585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:13.453591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:13.454965 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.455309 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.455310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.455311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.455360 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.455521 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.455521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.455522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.455647 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:13.455719 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.455796 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.455853 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:23:13.457433 18032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:23:13.458961 18032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:23:13.465634 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.4.47.3:34589)
0925 07:23:13.461930 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.461935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.461944 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:13.462047 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.462049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.462067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.462141 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.462414 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463481 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.464682 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465183 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465281 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465343 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.465511 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.465549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":85,"threads_started":1}
I20260925 07:23:13.465624 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34589 (local address 127.0.0.1:41964)
0925 07:23:13.461293 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.461334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.461396 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.462258 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.462259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.462262 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.462320 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.462322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463592 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.463593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.464590 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.464593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465286 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.465368 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465389 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.465493 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.465549 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260925 07:23:13.465852 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:32819 (local address 127.0.0.1:55236)
0925 07:23:13.461649 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.461658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.461706 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.462005 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.462009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.462011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.462079 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.462082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463469 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.463470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.464588 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.464591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465583 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.465656 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465663 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.465712 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.465759 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":12,"thread_start_us":54,"threads_started":1}
I20260925 07:23:13.465963 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:45969 (local address 127.0.0.1:44680)
0925 07:23:13.461768 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.461774 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.461806 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.462094 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.462095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.462096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.462165 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.462166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463180 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.463181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.464589 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.464591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465585 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.465656 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465664 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.465850 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.465893 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":8,"thread_start_us":47,"threads_started":1}
I20260925 07:23:13.466002 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.4.47.2:45969)
0925 07:23:13.461674 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.461677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.461855 (+ 178us) server_negotiation.cc:407] Connection header received
0925 07:23:13.461908 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.461911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.461940 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.462018 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.462317 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463047 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.464693 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465419 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465491 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465545 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.465884 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.465929 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 07:23:13.466102 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.461245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.4.47.1:32819)
0925 07:23:13.461456 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.461462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.461721 (+ 259us) server_negotiation.cc:407] Connection header received
0925 07:23:13.461771 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.461788 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.461805 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.461897 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.462164 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.463353 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.464692 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.465419 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.465491 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.465545 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.465954 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.466002 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260925 07:23:13.466745 18291 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (DEFAULT_TABLE table=test-table [id=98214a01a67841d98cfee0da84c088a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:13.466974 18157 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (DEFAULT_TABLE table=test-table [id=98214a01a67841d98cfee0da84c088a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:13.467128 18291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb51f0ebd2443aa85abddf52f2c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:13.467293 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb51f0ebd2443aa85abddf52f2c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:13.467399 18425 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (DEFAULT_TABLE table=test-table [id=98214a01a67841d98cfee0da84c088a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:13.467680 18425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb51f0ebd2443aa85abddf52f2c97d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:13.469727 18505 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Bootstrap starting.
I20260925 07:23:13.470396 18505 tablet_bootstrap.cc:654] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:13.470541 18506 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Bootstrap starting.
I20260925 07:23:13.470707 18507 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Bootstrap starting.
I20260925 07:23:13.470971 18505 log.cc:824] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:13.471367 18506 tablet_bootstrap.cc:654] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:13.471385 18507 tablet_bootstrap.cc:654] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:13.471875 18505 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: No bootstrap required, opened a new log
I20260925 07:23:13.471871 18507 log.cc:824] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:13.471922 18506 log.cc:824] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:13.471953 18505 ts_tablet_manager.cc:1397] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:13.472549 18507 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: No bootstrap required, opened a new log
I20260925 07:23:13.472584 18506 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: No bootstrap required, opened a new log
I20260925 07:23:13.472632 18507 ts_tablet_manager.cc:1397] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:23:13.472637 18506 ts_tablet_manager.cc:1397] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:13.473932 18507 raft_consensus.cc:348] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.474095 18507 raft_consensus.cc:374] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:13.473991 18505 raft_consensus.cc:348] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.474148 18505 raft_consensus.cc:374] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:13.474162 18507 raft_consensus.cc:729] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694befd00ac4026968dc07322ee45cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:13.474244 18505 raft_consensus.cc:729] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b5aab706444090a161b861078359cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:13.474279 18507 consensus_queue.cc:260] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.474373 18505 consensus_queue.cc:260] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.474591 18507 ts_tablet_manager.cc:1428] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:13.474654 18505 ts_tablet_manager.cc:1428] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:23:13.474588 18506 raft_consensus.cc:348] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.474727 18223 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
I20260925 07:23:13.474776 18506 raft_consensus.cc:374] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:13.474820 18506 raft_consensus.cc:729] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5864ecc46254d9f9c0a9407d053a62e, State: Initialized, Role: FOLLOWER
I20260925 07:23:13.474982 18491 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
I20260925 07:23:13.474934 18506 consensus_queue.cc:260] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.475265 18506 ts_tablet_manager.cc:1428] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 07:23:13.475412 18357 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
W20260925 07:23:13.519456 18358 tablet.cc:2369] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:13.542797 18512 raft_consensus.cc:482] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:13.542977 18512 raft_consensus.cc:504] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.543473 18512 leader_election.cc:288] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:13.546505 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.543789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:59081 (local address 127.4.47.3:34589)
0925 07:23:13.543836 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.543838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.543848 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:23:13.543936 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.543936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.543947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.543993 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.544189 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.544665 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.545680 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.546062 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.546175 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.546187 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.546382 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.546423 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:23:13.546514 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.543908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46747 (local address 127.4.47.2:45969)
0925 07:23:13.543958 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.543960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.544057 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:23:13.544122 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.544123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.544135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.544179 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.544352 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.544803 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.545746 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.546154 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.546220 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.546230 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.546402 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.546442 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:23:13.546540 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.543697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34589 (local address 127.4.47.1:59081)
0925 07:23:13.543768 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.543783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.543864 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.544013 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.544014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.544016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.544095 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.544096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.544763 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.544764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.545573 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.545575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.546178 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.546179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.546289 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.546295 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.546363 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.546423 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":23}
I20260925 07:23:13.546550 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.543817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:45969 (local address 127.4.47.1:46747)
0925 07:23:13.544003 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.544013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.544062 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.544180 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.544181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.544182 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.544250 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.544251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.544898 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.544899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.545647 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.545649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.546220 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.546221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.546316 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.546320 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.546398 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.546439 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":138,"threads_started":1}
I20260925 07:23:13.546998 18445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb51f0ebd2443aa85abddf52f2c97d4" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694befd00ac4026968dc07322ee45cf" is_pre_election: true
I20260925 07:23:13.547042 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb51f0ebd2443aa85abddf52f2c97d4" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5864ecc46254d9f9c0a9407d053a62e" is_pre_election: true
I20260925 07:23:13.547219 18445 raft_consensus.cc:2454] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 0.
I20260925 07:23:13.547282 18311 raft_consensus.cc:2454] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 0.
I20260925 07:23:13.547495 18110 leader_election.cc:302] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b5aab706444090a161b861078359cf, f694befd00ac4026968dc07322ee45cf; no voters:
I20260925 07:23:13.547662 18512 raft_consensus.cc:2781] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:13.547744 18512 raft_consensus.cc:482] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:13.547761 18512 raft_consensus.cc:3037] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:13.548581 18512 raft_consensus.cc:504] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.548724 18512 leader_election.cc:288] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 election: Requested vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:13.548992 18445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb51f0ebd2443aa85abddf52f2c97d4" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694befd00ac4026968dc07322ee45cf"
I20260925 07:23:13.548986 18311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb51f0ebd2443aa85abddf52f2c97d4" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5864ecc46254d9f9c0a9407d053a62e"
I20260925 07:23:13.549068 18445 raft_consensus.cc:3037] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:13.549088 18311 raft_consensus.cc:3037] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:13.549836 18311 raft_consensus.cc:2454] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 1.
I20260925 07:23:13.549894 18445 raft_consensus.cc:2454] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 1.
I20260925 07:23:13.550029 18110 leader_election.cc:302] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b5aab706444090a161b861078359cf, f694befd00ac4026968dc07322ee45cf; no voters:
I20260925 07:23:13.550141 18512 raft_consensus.cc:2781] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:13.550339 18512 raft_consensus.cc:686] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 LEADER]: Becoming Leader. State: Replica: 61b5aab706444090a161b861078359cf, State: Running, Role: LEADER
I20260925 07:23:13.550457 18512 consensus_queue.cc:237] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.551397 18030 catalog_manager.cc:5495] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b5aab706444090a161b861078359cf (127.4.47.1). New cstate: current_term: 1 leader_uuid: "61b5aab706444090a161b861078359cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } health_report { overall_health: UNKNOWN } } }
I20260925 07:23:13.559755 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.557531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.4.47.1:32819)
0925 07:23:13.557578 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.557580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.557590 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:23:13.557670 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.557671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.557689 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.557747 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:13.557853 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.558308 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.559004 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.559246 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.559298 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.559425 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:13.559522 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:13.559579 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.559610 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.559628 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:23:13.561597 18311 raft_consensus.cc:1261] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Refusing update from remote peer 61b5aab706444090a161b861078359cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:13.561597 18445 raft_consensus.cc:1261] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Refusing update from remote peer 61b5aab706444090a161b861078359cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:13.561975 18515 consensus_queue.cc:1035] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:13.562112 18512 consensus_queue.cc:1035] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:13.564618 18520 mvcc.cc:205] Tried to move back new op lower bound from 7333154789625360384 to 7333154789583372288. Current Snapshot: MvccSnapshot[applied={T|T < 7333154789625360384}]
I20260925 07:23:13.566633 18032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:13.566728 18032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:13.567469 18032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Sending DeleteTablet for 3 replicas of tablet 2eb51f0ebd2443aa85abddf52f2c97d4
I20260925 07:23:13.567785 18157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:13 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:55236
I20260925 07:23:13.567822 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18094
I20260925 07:23:13.567935 18425 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:13 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:41964
I20260925 07:23:13.568214 18291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:13 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44680
I20260925 07:23:13.568526 18527 tablet_replica.cc:330] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: stopping tablet replica
I20260925 07:23:13.568645 18528 tablet_replica.cc:330] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: stopping tablet replica
I20260925 07:23:13.568765 18528 raft_consensus.cc:2229] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:13.568887 18527 raft_consensus.cc:2229] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:13.569029 18528 raft_consensus.cc:2258] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:13.569211 18527 raft_consensus.cc:2258] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:13.569536 18528 ts_tablet_manager.cc:1905] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:13.569644 18527 ts_tablet_manager.cc:1905] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:13.571267 18527 ts_tablet_manager.cc:1918] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:13.571352 18527 log.cc:1196] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/2eb51f0ebd2443aa85abddf52f2c97d4
I20260925 07:23:13.571570 18528 ts_tablet_manager.cc:1918] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:13.571650 18528 log.cc:1196] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/2eb51f0ebd2443aa85abddf52f2c97d4
I20260925 07:23:13.571663 18527 ts_tablet_manager.cc:1939] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f694befd00ac4026968dc07322ee45cf: Deleting consensus metadata
I20260925 07:23:13.571956 18528 ts_tablet_manager.cc:1939] T 2eb51f0ebd2443aa85abddf52f2c97d4 P f5864ecc46254d9f9c0a9407d053a62e: Deleting consensus metadata
I20260925 07:23:13.572081 18016 catalog_manager.cc:4841] TS f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (table test-table [id=98214a01a67841d98cfee0da84c088a7]) successfully deleted
I20260925 07:23:13.572395 18016 catalog_manager.cc:4841] TS f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (table test-table [id=98214a01a67841d98cfee0da84c088a7]) successfully deleted
W20260925 07:23:13.576715 18018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:32819 (error 108)
I20260925 07:23:13.576869 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18228
I20260925 07:23:13.577601 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.577462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:32819 (local address 127.0.0.1:55250)
0925 07:23:13.577520 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.577559 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:23:13.578372 18018 catalog_manager.cc:4584] TS 61b5aab706444090a161b861078359cf (127.4.47.1:32819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2eb51f0ebd2443aa85abddf52f2c97d4: Network error: Client connection negotiation failed: client connection to 127.4.47.1:32819: connect: Connection refused (error 111)
I20260925 07:23:13.581915 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18362
I20260925 07:23:13.586172 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18000
I20260925 07:23:13.591230 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--webserver_interface=127.4.47.62
--webserver_port=36067
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:13.673892 18530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.674111 18530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.674134 18530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.675254 18530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:13.675290 18530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.675307 18530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.675319 18530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:13.675331 18530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:13.676714 18530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.676919 18530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.677090 18530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.685889 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.685891 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.685926 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.685925 18530 server_base.cc:1034] running on GCE node
I20260925 07:23:13.686630 18530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.686905 18530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.688069 18530 hybrid_clock.cc:648] HybridClock initialized: now 1790320993688061 us; error 30 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.690657 18530 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:13.691239 18530 webserver.cc:460] Webserver started at http://127.4.47.62:36067/ using document root <none> and password file <none>
I20260925 07:23:13.691496 18530 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.691578 18530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.692970 18530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:23:13.693682 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.693869 18530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:13.693971 18530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "ce49a948b52f4233a05408d6dbe9e0ca"
format_stamp: "Formatted at 2026-09-25 07:23:12 on dist-test-slave-zw07"
server_key: "ad77ddb435029f7d24fee8e68e95cdb4"
server_key_iv: "531e57a8e5982c101a01eaddb479dc9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.694417 18530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:13.703418 18530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:13.705854 18530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:13.706022 18530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:13.710141 18530 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33467
I20260925 07:23:13.710189 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33467 every 8 connection(s)
I20260925 07:23:13.710592 18530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:13.712216 18599 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:23:13.712807 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap starting.
I20260925 07:23:13.716166 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18530
I20260925 07:23:13.716526 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:32819
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=43885
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:13.721772 18599 log.cc:824] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:13.723865 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:13.724185 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap complete.
I20260925 07:23:13.726544 18599 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:13.726771 18599 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Initialized, Role: FOLLOWER
I20260925 07:23:13.726954 18599 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:13.727015 18599 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:13.727069 18599 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:13.727139 18599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:23:13.728160 18599 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:13.728250 18599 leader_election.cc:302] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce49a948b52f4233a05408d6dbe9e0ca; no voters:
I20260925 07:23:13.728482 18599 leader_election.cc:288] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:23:13.728518 18603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:23:13.728682 18603 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 LEADER]: Becoming Leader. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Running, Role: LEADER
I20260925 07:23:13.728760 18599 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:13.728828 18603 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:13.729108 18603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce49a948b52f4233a05408d6dbe9e0ca. Latest consensus state: current_term: 2 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:13.729202 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:13.729278 18603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:13.729367 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:13.729867 18612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:13.730826 18612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:13.730989 18612 catalog_manager.cc:1259] Loaded cluster ID: bc728b7f660844049a3c4452cb898d30
I20260925 07:23:13.731267 18612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:23:13.731506 18612 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 07:23:13.734186 18612 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 07:23:13.736025 18612 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:13.736231 18612 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Loaded TSK: 0
I20260925 07:23:13.736447 18612 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:23:13.831167 18601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.831459 18601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.831501 18601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.833267 18601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.833340 18601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.833384 18601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:13.835570 18601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:32819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=43885
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.835819 18601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.836107 18601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.841943 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.842023 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.842131 18601 server_base.cc:1034] running on GCE node
W20260925 07:23:13.842177 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.842485 18601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.842782 18601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.843979 18601 hybrid_clock.cc:648] HybridClock initialized: now 1790320993843954 us; error 41 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.846943 18601 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:13.847432 18601 webserver.cc:460] Webserver started at http://127.4.47.1:43885/ using document root <none> and password file <none>
I20260925 07:23:13.847673 18601 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.847757 18601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.849315 18601 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:13.850153 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.850337 18601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:13.850462 18601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "61b5aab706444090a161b861078359cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "ae715abaec34100c3bc3870898731e9c"
server_key_iv: "6e563441feb613054e27f6e1151da776"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.850795 18601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:13.866354 18601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:13.868845 18601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:13.869043 18601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:13.869302 18601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:13.869823 18643 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:23:13.871325 18601 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:23:13.871374 18601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:23:13.871410 18601 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:23:13.872081 18601 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:23:13.872125 18601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:23:13.872171 18643 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Bootstrap starting.
I20260925 07:23:13.879980 18601 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:32819
I20260925 07:23:13.880031 18750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:32819 every 8 connection(s)
I20260925 07:23:13.880584 18601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:13.882354 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18601
I20260925 07:23:13.882616 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:45969
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=43189
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:13.883698 18643 log.cc:824] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:13.886590 18643 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:13.886986 18643 tablet_bootstrap.cc:492] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Bootstrap complete.
I20260925 07:23:13.888278 18643 ts_tablet_manager.cc:1397] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:13.890112 18643 raft_consensus.cc:348] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.891266 18643 raft_consensus.cc:729] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b5aab706444090a161b861078359cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:13.891516 18643 consensus_queue.cc:260] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } } peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } }
I20260925 07:23:13.891880 18643 ts_tablet_manager.cc:1428] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 07:23:13.895140 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.881292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:56751)
0925 07:23:13.881476 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.881527 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.882263 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.884414 (+ 2151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.884418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.884783 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:13.885042 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.885049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.886706 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.886707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.888144 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.888147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.888973 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.888974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.889116 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.889721 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:13.889731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:13.891388 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:13.893497 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.893503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.893505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.893733 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.894002 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:13.894003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:13.894003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:13.894114 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:13.894493 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:13.894500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:13.894635 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.894743 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.894916 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260925 07:23:13.895155 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.881406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:56751 (local address 127.4.47.62:33467)
0925 07:23:13.883970 (+ 2564us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.883980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.884021 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:23:13.884161 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.884167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.884231 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.884386 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:13.885185 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.886573 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.888259 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.888849 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.889428 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.891559 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:13.891563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:13.891563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:13.891577 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:13.893388 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.893822 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.893826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.893826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.893924 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:13.894183 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:13.894184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:13.894184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:13.894411 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:13.894501 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.894831 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.894952 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2507,"thread_start_us":84,"threads_started":1}
I20260925 07:23:13.896203 18751 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:13.896378 18751 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:13.896574 18751 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:13.897374 18563 ts_manager.cc:194] Registered new tserver with Master: 61b5aab706444090a161b861078359cf (127.4.47.1:32819)
I20260925 07:23:13.898494 18563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:56751
I20260925 07:23:13.899511 18751 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
I20260925 07:23:13.902479 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.898220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.4.47.1:32819)
0925 07:23:13.899179 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:13.899184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:13.899446 (+ 262us) server_negotiation.cc:407] Connection header received
0925 07:23:13.899481 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:13.899483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:13.899496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:13.899563 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:13.899804 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.900556 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.901571 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.902035 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.902122 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.902177 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:13.902372 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 07:23:13.902409 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":79,"threads_started":1}
I20260925 07:23:13.902930 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:13.897939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:32819 (local address 127.0.0.1:55260)
0925 07:23:13.899248 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:13.899283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:13.899370 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:13.899623 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:13.899626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:13.899628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:13.899725 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.899728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.900656 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.900657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.901481 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:13.901483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:13.902122 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:13.902123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:13.902228 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:13.902242 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:23:13.902352 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 07:23:13.902409 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":100,"threads_started":1}
I20260925 07:23:13.903373 18685 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb51f0ebd2443aa85abddf52f2c97d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:13 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260925 07:23:13.903896 18760 tablet_replica.cc:330] stopping tablet replica
I20260925 07:23:13.904022 18760 raft_consensus.cc:2229] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:13.904093 18760 raft_consensus.cc:2258] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:13.904397 18760 ts_tablet_manager.cc:1905] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:13.906533 18760 ts_tablet_manager.cc:1918] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:13.906621 18760 log.cc:1196] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/2eb51f0ebd2443aa85abddf52f2c97d4
I20260925 07:23:13.906883 18760 ts_tablet_manager.cc:1939] T 2eb51f0ebd2443aa85abddf52f2c97d4 P 61b5aab706444090a161b861078359cf: Deleting consensus metadata
I20260925 07:23:13.907461 18549 catalog_manager.cc:4841] TS 61b5aab706444090a161b861078359cf (127.4.47.1:32819): tablet 2eb51f0ebd2443aa85abddf52f2c97d4 (table test-table [id=98214a01a67841d98cfee0da84c088a7]) successfully deleted
W20260925 07:23:13.973609 18756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:13.973817 18756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:13.973850 18756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:13.974926 18756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:13.974959 18756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:13.974985 18756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:13.976367 18756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:45969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=43189
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:13.976554 18756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:13.976732 18756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:13.981820 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.981802 18768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:13.981820 18766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:13.982088 18756 server_base.cc:1034] running on GCE node
I20260925 07:23:13.982316 18756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:13.982589 18756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:13.983757 18756 hybrid_clock.cc:648] HybridClock initialized: now 1790320993983756 us; error 49 us; skew 500 ppm
Sep 25 07:23:13 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:13.986258 18756 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:13.986763 18756 webserver.cc:460] Webserver started at http://127.4.47.2:43189/ using document root <none> and password file <none>
I20260925 07:23:13.986990 18756 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:13.987071 18756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:13.988469 18756 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:13.989092 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:13.989277 18756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:13.989405 18756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "f5864ecc46254d9f9c0a9407d053a62e"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "b11cb2a06a5eefb513e5a46044b85262"
server_key_iv: "0d9166ee5c0e086e5d78baaab57f25bf"
server_key_version: "encryptionkey@0"
I20260925 07:23:13.989763 18756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.002228 18756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.004189 18756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.004369 18756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.004616 18756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:14.004925 18756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:14.004978 18756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.005019 18756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:14.005046 18756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.011276 18756 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:45969
I20260925 07:23:14.011322 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:45969 every 8 connection(s)
I20260925 07:23:14.011677 18756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:14.018924 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18756
I20260925 07:23:14.019251 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:34589
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=39967
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:14.021507 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.012424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:35535 (local address 127.4.47.62:33467)
0925 07:23:14.012486 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.012489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.012691 (+ 202us) server_negotiation.cc:407] Connection header received
0925 07:23:14.013170 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.013171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.013188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.013251 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:14.014198 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.014711 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.015698 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.015903 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.015960 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.018145 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:14.018146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:14.018147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:14.018154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:14.019826 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.020361 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.020362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.020363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.020449 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.020766 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.020767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.020768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.020948 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:14.021021 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.021276 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.021373 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:23:14.021536 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.012408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:35535)
0925 07:23:14.012556 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.012599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.013086 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.013306 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.013310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.013905 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:14.014107 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.014112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.014803 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.014804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.015627 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.015629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.015982 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.015983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.016086 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.016567 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:14.016576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:14.018049 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:14.019918 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:14.019928 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:14.019930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:14.020261 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:14.020529 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:14.020530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:14.020530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:14.020698 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:14.021017 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:14.021022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:14.021147 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.021233 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.021363 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 07:23:14.022145 18889 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:14.022251 18889 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:14.022459 18889 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:14.022886 18563 ts_manager.cc:194] Registered new tserver with Master: f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:14.023340 18563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:35535
W20260925 07:23:14.103267 18892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:14.103511 18892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:14.103550 18892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:14.104676 18892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:14.104743 18892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:14.104789 18892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:14.106343 18892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:34589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=39967
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:14.106622 18892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:14.106863 18892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:14.111851 18898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.111851 18900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.111912 18897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.112195 18892 server_base.cc:1034] running on GCE node
I20260925 07:23:14.112416 18892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:14.112633 18892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:14.113775 18892 hybrid_clock.cc:648] HybridClock initialized: now 1790320994113764 us; error 33 us; skew 500 ppm
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:14.116426 18892 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:14.116883 18892 webserver.cc:460] Webserver started at http://127.4.47.3:39967/ using document root <none> and password file <none>
I20260925 07:23:14.117126 18892 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:14.117182 18892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:14.118604 18892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:14.119252 18907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.119464 18892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:14.119555 18892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "f694befd00ac4026968dc07322ee45cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "519720fc52187e4c52571093b01afe0e"
server_key_iv: "e9f4a974c465c054739525dcdd9a33ce"
server_key_version: "encryptionkey@0"
I20260925 07:23:14.119904 18892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.140275 18892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.142239 18892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.142408 18892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.142649 18892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:14.142961 18892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:14.142994 18892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.143018 18892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:14.143083 18892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.149492 18892 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:34589
I20260925 07:23:14.149536 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:34589 every 8 connection(s)
I20260925 07:23:14.149901 18892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:14.154989 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 18892
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:14.161201 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.150747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:57909 (local address 127.4.47.62:33467)
0925 07:23:14.150845 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.150847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.151037 (+ 190us) server_negotiation.cc:407] Connection header received
0925 07:23:14.151587 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.151589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.151600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.151657 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:14.153713 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.154315 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.155483 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.155697 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.155802 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.158098 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:14.158099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:14.158099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:14.158105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:14.159695 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.160163 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.160163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.160164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.160217 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.160521 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.160522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.160523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.160641 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:14.160720 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.161060 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.161132 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 07:23:14.161486 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.150719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:57909)
0925 07:23:14.150903 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.150941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.151488 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.151735 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.151739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.153332 (+ 1593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:14.153550 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.153557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.154421 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.154422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.155387 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.155390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.155821 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.155823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.155958 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.156547 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:14.156556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:14.158009 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:14.159769 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:14.159777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:14.159779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:14.160081 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:14.160290 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:14.160291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:14.160292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:14.160440 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:14.160710 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:14.160717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:14.160900 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.161031 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.161205 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
I20260925 07:23:14.161989 19021 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33467
I20260925 07:23:14.162110 19021 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:14.162312 19021 heartbeater.cc:511] Master 127.4.47.62:33467 requested a full tablet report, sending...
I20260925 07:23:14.162631 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.155535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.4.47.62:33467)
0925 07:23:14.155639 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.155641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.155736 (+ 95us) server_negotiation.cc:407] Connection header received
0925 07:23:14.155822 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.155823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.155834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.155896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:14.156624 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.157172 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.157954 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.158224 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.158278 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.159618 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:14.159619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:14.159619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:14.159625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:14.161415 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.161812 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.161813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.161814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.161876 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:14.162074 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:14.162075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:14.162075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:14.162209 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:14.162407 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.162444 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.162524 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260925 07:23:14.162806 18563 ts_manager.cc:194] Registered new tserver with Master: f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589)
I20260925 07:23:14.163296 18563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:57909
I20260925 07:23:14.163347 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:14.166050 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.163665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.4.47.62:33467)
0925 07:23:14.163703 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.163705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.163860 (+ 155us) server_negotiation.cc:407] Connection header received
0925 07:23:14.163895 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.163896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.163923 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.163973 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:14.164106 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.164548 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.165270 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.165551 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.165626 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.165796 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:14.165886 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:14.165952 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.165987 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.166005 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:23:14.166352 18563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:23:14.168004 18563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:23:14.170083 18685 tablet_service.cc:1467] Processing CreateTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 (DEFAULT_TABLE table=test-table [id=7afb928162ba4952bc082e7361bae91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:14.170310 18685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b5c8473aa54b469a61a215eb5aadf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:14.171814 18643 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Bootstrap starting.
I20260925 07:23:14.172154 18643 tablet_bootstrap.cc:654] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:14.173416 18643 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: No bootstrap required, opened a new log
I20260925 07:23:14.173496 18643 ts_tablet_manager.cc:1397] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260925 07:23:14.173614 18643 raft_consensus.cc:348] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.173694 18643 raft_consensus.cc:374] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:14.173722 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.169784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34589 (local address 127.0.0.1:41976)
0925 07:23:14.169878 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.169891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.169931 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.170442 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.170444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.170445 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:14.170488 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.170489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.171581 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.171583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.172608 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.172611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.173481 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.173482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.173551 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.173555 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.173603 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.173646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 07:23:14.173745 18643 raft_consensus.cc:729] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b5aab706444090a161b861078359cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:14.173799 18643 consensus_queue.cc:260] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.173931 18643 ts_tablet_manager.cc:1428] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.174122 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.170012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.4.47.3:34589)
0925 07:23:14.170214 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.170219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.170231 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:14.170270 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.170272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.170288 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.170375 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:14.170570 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.171487 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.172699 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.173388 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.173464 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.173632 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.173660 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.174041 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260925 07:23:14.174772 18955 tablet_service.cc:1467] Processing CreateTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 (DEFAULT_TABLE table=test-table [id=7afb928162ba4952bc082e7361bae91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:14.175141 18955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b5c8473aa54b469a61a215eb5aadf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:14.175458 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.169895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:45969 (local address 127.0.0.1:44688)
0925 07:23:14.171864 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.171875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.171927 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.172151 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.172152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.172154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:14.172197 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.172199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.173085 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.173147 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.174244 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.174248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.175155 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.175156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.175240 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.175244 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.175348 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.175392 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1950,"thread_start_us":89,"threads_started":1}
I20260925 07:23:14.175500 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.170012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.4.47.2:45969)
0925 07:23:14.170511 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.170516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.171988 (+ 1472us) server_negotiation.cc:407] Connection header received
0925 07:23:14.172034 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.172042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.172056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.172144 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:14.172255 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.172974 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.174362 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.174923 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.175012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.175110 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.175371 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.175417 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":100,"threads_started":1}
I20260925 07:23:14.176229 18823 tablet_service.cc:1467] Processing CreateTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 (DEFAULT_TABLE table=test-table [id=7afb928162ba4952bc082e7361bae91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:14.176584 18823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b5c8473aa54b469a61a215eb5aadf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:14.179239 19031 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Bootstrap starting.
I20260925 07:23:14.179841 19031 tablet_bootstrap.cc:654] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:14.179903 19032 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Bootstrap starting.
I20260925 07:23:14.180442 19032 tablet_bootstrap.cc:654] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:14.180516 19031 log.cc:824] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:14.181015 19032 log.cc:824] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:14.181322 19031 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: No bootstrap required, opened a new log
I20260925 07:23:14.181432 19031 ts_tablet_manager.cc:1397] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:23:14.181732 19032 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: No bootstrap required, opened a new log
I20260925 07:23:14.181807 19032 ts_tablet_manager.cc:1397] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:23:14.182983 19031 raft_consensus.cc:348] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.183207 19031 raft_consensus.cc:374] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:14.183256 19031 raft_consensus.cc:729] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f694befd00ac4026968dc07322ee45cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:14.183382 19031 consensus_queue.cc:260] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.183384 19032 raft_consensus.cc:348] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.183573 19032 raft_consensus.cc:374] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:14.183686 19021 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
I20260925 07:23:14.183691 19031 ts_tablet_manager.cc:1428] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:23:14.183619 19032 raft_consensus.cc:729] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5864ecc46254d9f9c0a9407d053a62e, State: Initialized, Role: FOLLOWER
I20260925 07:23:14.184051 19032 consensus_queue.cc:260] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.184392 18889 heartbeater.cc:503] Master 127.4.47.62:33467 was elected leader, sending a full tablet report...
I20260925 07:23:14.184490 19032 ts_tablet_manager.cc:1428] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 07:23:14.262339 18890 tablet.cc:2369] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:14.355686 18757 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:14.355824 18757 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.356261 18757 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:14.360056 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.356552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:60219 (local address 127.4.47.3:34589)
0925 07:23:14.356625 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.356627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.356635 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:23:14.356661 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.356662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.356671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.356762 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:14.356973 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.357679 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359022 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.359685 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359768 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.359778 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.359918 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.359970 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:23:14.360049 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.356449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34589 (local address 127.4.47.1:60219)
0925 07:23:14.356506 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.356519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.356571 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.356799 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.356801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.356802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:14.356866 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.356867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.357762 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.357763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.358925 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.358927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.359778 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.359833 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.359837 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.359909 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.359951 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":9}
I20260925 07:23:14.360144 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.356774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:43201 (local address 127.4.47.2:45969)
0925 07:23:14.356821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.356823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.356830 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:23:14.356861 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.356861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.356870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.356913 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:14.357349 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.357959 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359274 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.359830 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.359926 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.360035 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.360073 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:23:14.360215 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.356603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:45969 (local address 127.4.47.1:43201)
0925 07:23:14.356720 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.356731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.356776 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.356937 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:14.356938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:14.356939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:14.357242 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.357244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.358032 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.358032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.359200 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:14.359203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.359915 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.359916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:14.359965 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.359968 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:23:14.360019 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:23:14.360047 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260925 07:23:14.360508 18975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b5c8473aa54b469a61a215eb5aadf8" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694befd00ac4026968dc07322ee45cf" is_pre_election: true
I20260925 07:23:14.360538 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b5c8473aa54b469a61a215eb5aadf8" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5864ecc46254d9f9c0a9407d053a62e" is_pre_election: true
I20260925 07:23:14.360667 18975 raft_consensus.cc:2454] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 0.
I20260925 07:23:14.360723 18843 raft_consensus.cc:2454] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 0.
I20260925 07:23:14.360924 18637 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b5aab706444090a161b861078359cf, f694befd00ac4026968dc07322ee45cf; no voters:
I20260925 07:23:14.361083 18757 raft_consensus.cc:2781] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:14.361145 18757 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:14.361162 18757 raft_consensus.cc:3037] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:14.361907 18757 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.362046 18757 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 election: Requested vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
I20260925 07:23:14.362222 18975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b5c8473aa54b469a61a215eb5aadf8" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f694befd00ac4026968dc07322ee45cf"
I20260925 07:23:14.362250 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b5c8473aa54b469a61a215eb5aadf8" candidate_uuid: "61b5aab706444090a161b861078359cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5864ecc46254d9f9c0a9407d053a62e"
I20260925 07:23:14.362305 18975 raft_consensus.cc:3037] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:14.362383 18843 raft_consensus.cc:3037] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:14.363039 18975 raft_consensus.cc:2454] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 1.
I20260925 07:23:14.363147 18843 raft_consensus.cc:2454] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b5aab706444090a161b861078359cf in term 1.
I20260925 07:23:14.363245 18637 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b5aab706444090a161b861078359cf, f694befd00ac4026968dc07322ee45cf; no voters:
I20260925 07:23:14.363345 18757 raft_consensus.cc:2781] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:14.363497 18757 raft_consensus.cc:686] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 LEADER]: Becoming Leader. State: Replica: 61b5aab706444090a161b861078359cf, State: Running, Role: LEADER
I20260925 07:23:14.363584 18757 consensus_queue.cc:237] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.364342 18563 catalog_manager.cc:5495] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 61b5aab706444090a161b861078359cf (127.4.47.1). New cstate: current_term: 1 leader_uuid: "61b5aab706444090a161b861078359cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } health_report { overall_health: HEALTHY } } }
W20260925 07:23:14.392735 18752 tablet.cc:2369] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:14.400461 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.398289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.4.47.1:32819)
0925 07:23:14.398355 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:14.398357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:14.398369 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:14.398416 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:14.398416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:14.398435 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:14.398498 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:14.398614 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.399104 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.399771 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:14.399961 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:14.400011 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:14.400148 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:14.400243 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:14.400296 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:14.400339 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 07:23:14.400383 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 07:23:14.400513 19022 tablet.cc:2369] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:14.402155 18975 raft_consensus.cc:1261] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Refusing update from remote peer 61b5aab706444090a161b861078359cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:14.402155 18843 raft_consensus.cc:1261] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Refusing update from remote peer 61b5aab706444090a161b861078359cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:14.402563 19038 consensus_queue.cc:1035] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:14.402705 18757 consensus_queue.cc:1035] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:14.404893 19040 mvcc.cc:205] Tried to move back new op lower bound from 7333154793068122112 to 7333154792913842176. Current Snapshot: MvccSnapshot[applied={T|T < 7333154793068122112}]
I20260925 07:23:14.405210 19041 mvcc.cc:205] Tried to move back new op lower bound from 7333154793068122112 to 7333154792913842176. Current Snapshot: MvccSnapshot[applied={T|T < 7333154793068122112}]
I20260925 07:23:14.405718 19042 mvcc.cc:205] Tried to move back new op lower bound from 7333154793068122112 to 7333154792913842176. Current Snapshot: MvccSnapshot[applied={T|T < 7333154793068122112}]
I20260925 07:23:14.407157 18563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:14.407261 18563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:14.408017 18563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Sending DeleteTablet for 3 replicas of tablet f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:14.408394 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18601
I20260925 07:23:14.408411 18685 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:14 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260925 07:23:14.408388 18955 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:14 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:41976
I20260925 07:23:14.408455 18823 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b5c8473aa54b469a61a215eb5aadf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:14 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:44688
I20260925 07:23:14.409061 19051 tablet_replica.cc:330] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: stopping tablet replica
I20260925 07:23:14.409168 19051 raft_consensus.cc:2229] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:14.409253 19051 raft_consensus.cc:2258] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:14.409427 19050 tablet_replica.cc:330] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: stopping tablet replica
I20260925 07:23:14.409520 19050 raft_consensus.cc:2229] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:14.409595 19050 raft_consensus.cc:2258] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:14.409830 19051 ts_tablet_manager.cc:1905] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:14.409969 19050 ts_tablet_manager.cc:1905] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:14.411598 19050 ts_tablet_manager.cc:1918] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:14.411712 19050 log.cc:1196] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:14.411768 19051 ts_tablet_manager.cc:1918] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:14.411851 19051 log.cc:1196] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:14.412027 19050 ts_tablet_manager.cc:1939] T f3b5c8473aa54b469a61a215eb5aadf8 P f694befd00ac4026968dc07322ee45cf: Deleting consensus metadata
I20260925 07:23:14.412151 19051 ts_tablet_manager.cc:1939] T f3b5c8473aa54b469a61a215eb5aadf8 P f5864ecc46254d9f9c0a9407d053a62e: Deleting consensus metadata
I20260925 07:23:14.412573 18547 catalog_manager.cc:4841] TS f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): tablet f3b5c8473aa54b469a61a215eb5aadf8 (table test-table [id=7afb928162ba4952bc082e7361bae91e]) successfully deleted
I20260925 07:23:14.412708 18547 catalog_manager.cc:4841] TS f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): tablet f3b5c8473aa54b469a61a215eb5aadf8 (table test-table [id=7afb928162ba4952bc082e7361bae91e]) successfully deleted
W20260925 07:23:14.414781 18549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:32819 (error 108)
I20260925 07:23:14.415310 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.415211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:32819 (local address 127.0.0.1:45412)
0925 07:23:14.415252 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.415283 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 07:23:14.415462 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18756
W20260925 07:23:14.415489 18549 catalog_manager.cc:4584] TS 61b5aab706444090a161b861078359cf (127.4.47.1:32819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3b5c8473aa54b469a61a215eb5aadf8: Network error: Client connection negotiation failed: client connection to 127.4.47.1:32819: connect: Connection refused (error 111)
I20260925 07:23:14.421264 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18892
I20260925 07:23:14.426025 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 18530
I20260925 07:23:14.431931 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--webserver_interface=127.4.47.62
--webserver_port=36067
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:14.513047 19053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:14.513315 19053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:14.513357 19053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:14.514541 19053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:14.514613 19053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:14.514638 19053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:14.514659 19053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:14.514689 19053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:14.516228 19053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:14.516527 19053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:14.516758 19053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:14.521679 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.521687 19058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.521687 19059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.521756 19053 server_base.cc:1034] running on GCE node
I20260925 07:23:14.522184 19053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:14.522501 19053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:14.523665 19053 hybrid_clock.cc:648] HybridClock initialized: now 1790320994523645 us; error 57 us; skew 500 ppm
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:14.526120 19053 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:14.526582 19053 webserver.cc:460] Webserver started at http://127.4.47.62:36067/ using document root <none> and password file <none>
I20260925 07:23:14.526823 19053 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:14.526870 19053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:14.528309 19053 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:14.529054 19068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.529264 19053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:14.529356 19053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "ce49a948b52f4233a05408d6dbe9e0ca"
format_stamp: "Formatted at 2026-09-25 07:23:12 on dist-test-slave-zw07"
server_key: "ad77ddb435029f7d24fee8e68e95cdb4"
server_key_iv: "531e57a8e5982c101a01eaddb479dc9d"
server_key_version: "encryptionkey@0"
I20260925 07:23:14.529713 19053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.543828 19053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.545841 19053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.546027 19053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.550637 19053 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33467
I20260925 07:23:14.550665 19120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33467 every 8 connection(s)
I20260925 07:23:14.551126 19053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:14.552706 19121 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:23:14.553303 19121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap starting.
I20260925 07:23:14.556872 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19053
I20260925 07:23:14.557148 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:32819
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=43885
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:14.561626 19121 log.cc:824] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:14.564007 19121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:14.564343 19121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca: Bootstrap complete.
I20260925 07:23:14.567322 19121 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:14.567615 19121 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Initialized, Role: FOLLOWER
I20260925 07:23:14.567881 19121 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:14.567996 19121 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:14.568055 19121 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:14.568104 19121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:23:14.569167 19121 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:14.569293 19121 leader_election.cc:302] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce49a948b52f4233a05408d6dbe9e0ca; no voters:
I20260925 07:23:14.569487 19121 leader_election.cc:288] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 07:23:14.569559 19125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 3 FOLLOWER]: Leader election won for term 3
I20260925 07:23:14.569707 19125 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [term 3 LEADER]: Becoming Leader. State: Replica: ce49a948b52f4233a05408d6dbe9e0ca, State: Running, Role: LEADER
I20260925 07:23:14.569772 19121 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:14.569818 19125 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } }
I20260925 07:23:14.570098 19125 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce49a948b52f4233a05408d6dbe9e0ca. Latest consensus state: current_term: 3 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:14.570169 19125 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:14.570101 19126 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49a948b52f4233a05408d6dbe9e0ca" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33467 } } }
I20260925 07:23:14.570361 19130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:14.570379 19126 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49a948b52f4233a05408d6dbe9e0ca [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:14.571275 19130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:14.571380 19130 catalog_manager.cc:1259] Loaded cluster ID: bc728b7f660844049a3c4452cb898d30
I20260925 07:23:14.571425 19130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 07:23:14.571869 19130 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fece81ba9ad google::LogMessage::Fail() at ??:0
@ 0x7fece81bd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fece81ba4de google::LogMessage::Flush() at ??:0
@ 0x7fece81bba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fecf00db0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fecf00e48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fecea0880d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fecea07d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feceaee16db start_thread at ??:0
@ 0x7fece6cb861f clone at ??:0
W20260925 07:23:14.651213 19123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:14.651465 19123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:14.651505 19123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:14.652733 19123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:14.652805 19123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:14.652853 19123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:14.654377 19123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:32819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=43885
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:14.654657 19123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:14.654894 19123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:14.660123 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.660135 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.660290 19123 server_base.cc:1034] running on GCE node
W20260925 07:23:14.660132 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.660758 19123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:14.661051 19123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:14.662248 19123 hybrid_clock.cc:648] HybridClock initialized: now 1790320994662234 us; error 28 us; skew 500 ppm
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:14.665184 19123 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:14.665654 19123 webserver.cc:460] Webserver started at http://127.4.47.1:43885/ using document root <none> and password file <none>
I20260925 07:23:14.665859 19123 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:14.665910 19123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:14.667464 19123 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:14.668171 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.668327 19123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:14.668416 19123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "61b5aab706444090a161b861078359cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "ae715abaec34100c3bc3870898731e9c"
server_key_iv: "6e563441feb613054e27f6e1151da776"
server_key_version: "encryptionkey@0"
I20260925 07:23:14.668876 19123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.683364 19123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.685953 19123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.686074 19123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.686331 19123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:14.686863 19174 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:23:14.688400 19123 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:23:14.688462 19123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:23:14.688498 19123 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:23:14.689496 19123 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:23:14.689546 19123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:23:14.689687 19174 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Bootstrap starting.
I20260925 07:23:14.696944 19123 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:32819
I20260925 07:23:14.696993 19281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:32819 every 8 connection(s)
I20260925 07:23:14.697418 19123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:14.701814 19174 log.cc:824] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:14.702420 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19123
I20260925 07:23:14.702713 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:45969
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=43189
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:14.705642 19174 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:14.705938 19174 tablet_bootstrap.cc:492] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Bootstrap complete.
I20260925 07:23:14.712949 19174 ts_tablet_manager.cc:1397] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.008s
I20260925 07:23:14.714623 19174 raft_consensus.cc:348] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.714955 19174 raft_consensus.cc:729] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b5aab706444090a161b861078359cf, State: Initialized, Role: FOLLOWER
I20260925 07:23:14.715128 19174 consensus_queue.cc:260] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:14.715437 19174 ts_tablet_manager.cc:1428] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260925 07:23:14.792538 19284 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:23:14.700013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:55201)
0925 07:23:14.701623 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.701681 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:14.713833 (+ 12152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:14.792324 (+ 78491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1553,"thread_start_us":154,"threads_started":1}
W20260925 07:23:14.792796 19282 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33467 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33467: Client connection negotiation failed: client connection to 127.4.47.62:33467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:23:14.793169 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.793054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:47561)
0925 07:23:14.793111 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.793150 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:23:14.793500 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.793405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:60119)
0925 07:23:14.793452 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.793476 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:23:14.793602 19282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:23:14.800289 19286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:14.800501 19286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:14.800527 19286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:14.801699 19286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:14.801743 19286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:14.801770 19286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:14.803393 19286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:45969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=43189
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:14.803653 19286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:14.803891 19286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:14.809114 19295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.809154 19292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.809121 19293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.809584 19286 server_base.cc:1034] running on GCE node
I20260925 07:23:14.809809 19286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:14.810070 19286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:14.811221 19286 hybrid_clock.cc:648] HybridClock initialized: now 1790320994811206 us; error 27 us; skew 500 ppm
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:14.813966 19286 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:14.814631 19286 webserver.cc:460] Webserver started at http://127.4.47.2:43189/ using document root <none> and password file <none>
I20260925 07:23:14.814929 19286 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:14.815007 19286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:14.816437 19286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:14.817070 19302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.817250 19286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:14.817358 19286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "f5864ecc46254d9f9c0a9407d053a62e"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "b11cb2a06a5eefb513e5a46044b85262"
server_key_iv: "0d9166ee5c0e086e5d78baaab57f25bf"
server_key_version: "encryptionkey@0"
I20260925 07:23:14.817783 19286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.830256 19286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.832731 19286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.832895 19286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.833137 19286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:14.833454 19286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:14.833510 19286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.833551 19286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:14.833580 19286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.839599 19286 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:45969
I20260925 07:23:14.839658 19415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:45969 every 8 connection(s)
I20260925 07:23:14.840042 19286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:23:14.841178 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.840815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:36381)
0925 07:23:14.840964 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.841036 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260925 07:23:14.841620 19416 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33467 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33467: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
I20260925 07:23:14.841944 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.841821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:51535)
0925 07:23:14.841892 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.841924 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 07:23:14.842329 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.842172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:60449)
0925 07:23:14.842286 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.842312 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 07:23:14.842458 19416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:23:14.846304 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19286
I20260925 07:23:14.846535 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:34589
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:33467
--webserver_port=39967
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:14.929272 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:14.929487 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:14.929512 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:14.930662 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:14.930706 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:14.930732 19419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:14.932248 19419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:34589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=39967
--tserver_master_addrs=127.4.47.62:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:14.932499 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:14.932729 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:14.937614 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.937620 19424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:14.937637 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:14.937911 19419 server_base.cc:1034] running on GCE node
I20260925 07:23:14.938124 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:14.938313 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:14.939476 19419 hybrid_clock.cc:648] HybridClock initialized: now 1790320994939457 us; error 32 us; skew 500 ppm
Sep 25 07:23:14 dist-test-slave-zw07 krb5kdc[17987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:14.941843 19419 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:14.942282 19419 webserver.cc:460] Webserver started at http://127.4.47.3:39967/ using document root <none> and password file <none>
I20260925 07:23:14.942520 19419 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:14.942591 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:14.944029 19419 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:14.944609 19434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.944780 19419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:14.944880 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "f694befd00ac4026968dc07322ee45cf"
format_stamp: "Formatted at 2026-09-25 07:23:13 on dist-test-slave-zw07"
server_key: "519720fc52187e4c52571093b01afe0e"
server_key_iv: "e9f4a974c465c054739525dcdd9a33ce"
server_key_version: "encryptionkey@0"
I20260925 07:23:14.945226 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:14.968987 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:14.970960 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:14.971167 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:14.971453 19419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:14.971804 19419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:14.971863 19419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.971904 19419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:14.971953 19419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:14.977824 19419 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:34589
I20260925 07:23:14.977911 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:34589 every 8 connection(s)
I20260925 07:23:14.978278 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:14.979390 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.978957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:41485)
0925 07:23:14.979171 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.979247 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
W20260925 07:23:14.979810 19548 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33467 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33467: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
I20260925 07:23:14.980185 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.980098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:55389)
0925 07:23:14.980141 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.980168 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:23:14.980562 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:14.980488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:47817)
0925 07:23:14.980527 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:14.980548 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 07:23:14.980690 19548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:23:14.981436 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19419
I20260925 07:23:15.794289 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.794016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:45171)
0925 07:23:15.794179 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:15.794225 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260925 07:23:15.842984 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.842778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:33857)
0925 07:23:15.842895 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:15.842931 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260925 07:23:15.964854 19554 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:15.964973 19554 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:15.965269 19554 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
Sep 25 07:23:15 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.3@KRBTEST.COM
Sep 25 07:23:15 dist-test-slave-zw07 krb5kdc[17987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790320994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.2@KRBTEST.COM
I20260925 07:23:15.977883 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.965496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34589 (local address 127.4.47.1:41395)
0925 07:23:15.965553 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:15.965578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:15.965645 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:15.966726 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:15.966729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:15.967240 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:15.967528 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:15.967537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.969197 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.969199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:15.970541 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:15.970544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.971490 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.971492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:15.971669 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:15.972328 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:15.972332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:15.974147 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:15.976261 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:15.976271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:15.976499 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:15.976774 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:15.976775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:15.976887 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:15.977294 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:15.977301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:15.977446 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 07:23:15.977558 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 07:23:15.977733 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":56}
I20260925 07:23:15.977893 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.965637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:41395 (local address 127.4.47.3:34589)
0925 07:23:15.965778 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:15.965791 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:15.965824 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:23:15.966404 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:15.966407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:15.966484 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:15.966653 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:15.967658 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.969073 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.970643 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.971306 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.971447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:15.974264 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:15.974269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:15.974269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:15.974283 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:15.976155 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976596 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:15.976599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:15.976600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:15.976695 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976962 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:15.976963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:15.976964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:15.977218 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:15.977315 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:15.977617 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 07:23:15.977739 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260925 07:23:15.978032 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.965786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:32927 (local address 127.4.47.2:45969)
0925 07:23:15.965954 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:15.965961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:15.965990 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:23:15.966498 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:15.966501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:15.966539 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:15.966653 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:15.967657 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.969073 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.970635 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.971306 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.971447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:15.974725 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:15.974730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:15.974730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:15.974744 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:15.976508 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976897 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:15.976901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:15.976901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:15.976990 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:15.977224 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:15.977225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:15.977225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:15.977457 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:15.977546 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:15.977785 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 07:23:15.977933 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260925 07:23:15.979176 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:15.979354 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.965619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:45969 (local address 127.4.47.1:32927)
0925 07:23:15.965802 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:15.965812 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:15.965858 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:15.966726 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:15.966729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:15.967240 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:15.967528 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:15.967537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.969211 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.969211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:15.970541 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:15.970544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:15.971492 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:15.971493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:15.971669 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:15.972320 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:15.972332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:15.974622 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:15.976620 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:15.976621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:15.976621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:15.976802 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:15.977062 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:15.977063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:15.977063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:15.977125 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:15.977531 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:15.977532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:15.977579 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:23:15.977673 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 07:23:15.979200 (+ 1527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":66,"thread_start_us":44,"threads_started":1}
W20260925 07:23:15.979974 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:15.980063 19168 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b5aab706444090a161b861078359cf; no voters: f5864ecc46254d9f9c0a9407d053a62e, f694befd00ac4026968dc07322ee45cf
I20260925 07:23:15.980207 19554 raft_consensus.cc:2726] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:23:15.981143 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:15.980957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:60001)
0925 07:23:15.981070 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:15.981108 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 07:23:16.795028 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:16.794815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:58833)
0925 07:23:16.794943 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:16.794978 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260925 07:23:16.843767 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:16.843512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:50921)
0925 07:23:16.843673 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:16.843718 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260925 07:23:16.981834 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:16.981614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:48931)
0925 07:23:16.981737 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:16.981776 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 07:23:17.730506 19562 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:17.730607 19562 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:17.730738 19562 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
W20260925 07:23:17.731387 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
W20260925 07:23:17.731478 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:17.731505 19168 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b5aab706444090a161b861078359cf; no voters: f5864ecc46254d9f9c0a9407d053a62e, f694befd00ac4026968dc07322ee45cf
I20260925 07:23:17.731573 19562 raft_consensus.cc:2726] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:23:17.795920 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:17.795624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:38589)
0925 07:23:17.795814 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:17.795866 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260925 07:23:17.844466 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:17.844250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:49855)
0925 07:23:17.844383 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:17.844423 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 07:23:17.982616 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:17.982385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:36037)
0925 07:23:17.982533 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:17.982573 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260925 07:23:18.796761 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:18.796529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:49635)
0925 07:23:18.796669 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:18.796712 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260925 07:23:18.845139 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:18.844938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:42637)
0925 07:23:18.845061 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:18.845100 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260925 07:23:18.983438 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:18.983216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:55091)
0925 07:23:18.983338 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:18.983382 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260925 07:23:19.797560 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:19.797342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:49259)
0925 07:23:19.797472 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:19.797513 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260925 07:23:19.845937 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:19.845643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:40661)
0925 07:23:19.845845 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:19.845891 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":62,"threads_started":1}
I20260925 07:23:19.984124 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:19.983885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:36321)
0925 07:23:19.984032 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:19.984076 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260925 07:23:20.798456 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:20.798170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:54591)
0925 07:23:20.798340 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:20.798387 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260925 07:23:20.836061 19573 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:20.836158 19573 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:20.836330 19573 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
W20260925 07:23:20.836809 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
W20260925 07:23:20.836882 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:20.836897 19168 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b5aab706444090a161b861078359cf; no voters: f5864ecc46254d9f9c0a9407d053a62e, f694befd00ac4026968dc07322ee45cf
I20260925 07:23:20.837005 19573 raft_consensus.cc:2726] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:23:20.846648 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:20.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:45757)
0925 07:23:20.846571 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:20.846608 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260925 07:23:20.984879 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:20.984630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:55155)
0925 07:23:20.984791 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:20.984831 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260925 07:23:21.799270 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:21.799002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:45133)
0925 07:23:21.799182 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:21.799220 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260925 07:23:21.847505 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:21.847186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:51229)
0925 07:23:21.847361 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:21.847409 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260925 07:23:21.985636 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:21.985417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:53991)
0925 07:23:21.985548 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:21.985590 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260925 07:23:22.799965 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:22.799735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:60141)
0925 07:23:22.799887 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:22.799919 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260925 07:23:22.848356 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:22.848096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:58357)
0925 07:23:22.848248 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:22.848292 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260925 07:23:22.986438 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:22.986188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:43685)
0925 07:23:22.986345 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:22.986390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260925 07:23:23.128252 19582 raft_consensus.cc:482] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:23.128401 19582 raft_consensus.cc:504] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f694befd00ac4026968dc07322ee45cf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34589 } } peers { permanent_uuid: "f5864ecc46254d9f9c0a9407d053a62e" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 45969 } } peers { permanent_uuid: "61b5aab706444090a161b861078359cf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 32819 } }
I20260925 07:23:23.128628 19582 leader_election.cc:288] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589), f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969)
W20260925 07:23:23.129109 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f694befd00ac4026968dc07322ee45cf (127.4.47.3:34589): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
W20260925 07:23:23.129173 19168 leader_election.cc:341] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5864ecc46254d9f9c0a9407d053a62e (127.4.47.2:45969): Not found: Tablet not found: f3b5c8473aa54b469a61a215eb5aadf8
I20260925 07:23:23.129190 19168 leader_election.cc:302] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b5aab706444090a161b861078359cf; no voters: f5864ecc46254d9f9c0a9407d053a62e, f694befd00ac4026968dc07322ee45cf
I20260925 07:23:23.129300 19582 raft_consensus.cc:2726] T f3b5c8473aa54b469a61a215eb5aadf8 P 61b5aab706444090a161b861078359cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:23:23.800858 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:23.800560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:51621)
0925 07:23:23.800717 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:23.800763 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260925 07:23:23.849392 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:23.849052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:42515)
0925 07:23:23.849189 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:23.849241 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260925 07:23:23.987327 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:23.986983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:36581)
0925 07:23:23.987174 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:23.987243 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":126,"threads_started":1}
I20260925 07:23:24.801707 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:24.801476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:39715)
0925 07:23:24.801618 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:24.801658 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260925 07:23:24.850137 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:24.849923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:47599)
0925 07:23:24.850048 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:24.850090 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 07:23:24.988041 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:24.987768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:48485)
0925 07:23:24.987940 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:24.987993 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260925 07:23:25.802501 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:25.802219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:58439)
0925 07:23:25.802412 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:25.802454 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 07:23:25.850889 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:25.850621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:54135)
0925 07:23:25.850776 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:25.850827 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260925 07:23:25.988863 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:25.988631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:39137)
0925 07:23:25.988782 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:25.988819 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260925 07:23:26.803413 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:26.803183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:48243)
0925 07:23:26.803319 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:26.803365 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 07:23:26.851683 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:26.851443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:43155)
0925 07:23:26.851594 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:26.851635 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260925 07:23:26.989629 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:26.989417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:54175)
0925 07:23:26.989545 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:26.989581 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 07:23:27.804141 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:27.803887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:50713)
0925 07:23:27.804051 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:27.804092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 07:23:27.852525 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:27.852263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:58767)
0925 07:23:27.852430 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:27.852477 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260925 07:23:27.990406 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:27.990154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:48645)
0925 07:23:27.990312 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:27.990358 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260925 07:23:28.805004 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:28.804766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:45177)
0925 07:23:28.804908 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:28.804952 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260925 07:23:28.853281 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:28.853070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:51775)
0925 07:23:28.853192 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:28.853232 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260925 07:23:28.991202 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:28.990911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:48677)
0925 07:23:28.991041 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:28.991154 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260925 07:23:29.805995 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:29.805737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.1:51891)
0925 07:23:29.805903 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:29.805947 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":118,"threads_started":1}
I20260925 07:23:29.854175 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:29.853910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.2:53469)
0925 07:23:29.854071 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:29.854121 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260925 07:23:29.986292 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19123
I20260925 07:23:29.991405 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19286
I20260925 07:23:29.992468 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:29.992093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33467 (local address 127.4.47.3:49657)
0925 07:23:29.992317 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:29.992351 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":58,"threads_started":1}
I20260925 07:23:29.995651 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19419
2026-09-25T07:23:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18349 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:23:30 dist-test-slave-zw07 krb5kdc[19606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:23:30 dist-test-slave-zw07 krb5kdc[19606](info): set up 2 sockets
Sep 25 07:23:30 dist-test-slave-zw07 krb5kdc[19606](info): commencing operation
krb5kdc: starting...
W20260925 07:23:31.071296 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:23:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:23:31.190305 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:31.270920 19619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:31.271152 19619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:31.271173 19619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:31.272349 19619 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:23:31.272389 19619 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:23:31.272403 19619 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:23:31.272414 19619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:31.272477 19619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:31.272496 19619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:31.272518 19619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:31.272547 19619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:31.274026 19619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:31.274293 19619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:31.274549 19619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:31.279667 19625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.279753 19624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.279713 19619 server_base.cc:1034] running on GCE node
W20260925 07:23:31.279680 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.280133 19619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:31.280417 19619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:31.281607 19619 hybrid_clock.cc:648] HybridClock initialized: now 1790321011281595 us; error 30 us; skew 500 ppm
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:31.284343 19619 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:31.284847 19619 webserver.cc:460] Webserver started at http://127.4.47.62:39885/ using document root <none> and password file <none>
I20260925 07:23:31.285077 19619 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:31.285122 19619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:31.285271 19619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:31.286345 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "e0628f20f5374a798d577a13495caa7d"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "6b17412672b7e4c7e378d2fe8e7920f1"
server_key_iv: "027f0ad2047ac01098bd487eb34586fe"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.286723 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "e0628f20f5374a798d577a13495caa7d"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "6b17412672b7e4c7e378d2fe8e7920f1"
server_key_iv: "027f0ad2047ac01098bd487eb34586fe"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.288138 19619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:23:31.288985 19634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.289162 19619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:31.289263 19619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "e0628f20f5374a798d577a13495caa7d"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "6b17412672b7e4c7e378d2fe8e7920f1"
server_key_iv: "027f0ad2047ac01098bd487eb34586fe"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.289350 19619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:31.307420 19619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:31.309885 19619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:31.310065 19619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:31.314275 19619 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33747
I20260925 07:23:31.314313 19686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33747 every 8 connection(s)
I20260925 07:23:31.314685 19619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:31.315179 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19619
I20260925 07:23:31.315302 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:23:31.315537 4284 external_mini_cluster.cc:1428] Setting key 413d6b0c589dceedc952f8d4a4530adb
I20260925 07:23:31.315913 19687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:31.318892 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap starting.
I20260925 07:23:31.319648 19687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:31.319972 19687 log.cc:824] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:31.320770 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: No bootstrap required, opened a new log
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:31.323380 19687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:31.323577 19687 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:31.323633 19687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Initialized, Role: FOLLOWER
I20260925 07:23:31.323853 19687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:31.323948 19687 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:31.323975 19687 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:31.324052 19687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:31.324692 19687 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:31.324786 19687 leader_election.cc:302] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0628f20f5374a798d577a13495caa7d; no voters:
I20260925 07:23:31.324999 19687 leader_election.cc:288] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:23:31.325054 19692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:31.325243 19692 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 LEADER]: Becoming Leader. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Running, Role: LEADER
I20260925 07:23:31.325321 19687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:31.325388 19692 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:31.325729 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0628f20f5374a798d577a13495caa7d. Latest consensus state: current_term: 1 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:23:31.325734 19694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:23:31.325896 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:31.325904 19694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:31.326170 19699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:31.326740 19699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:31.327250 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.316085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.4.47.62:33747)
0925 07:23:31.316296 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.316305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.316330 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:23:31.317001 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.317009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.317060 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.317209 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:31.317863 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.318757 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.319626 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.320139 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.320267 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.323148 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:31.323152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:31.323152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:31.323163 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:31.325094 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.325558 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.325562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.325563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.325672 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.325938 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.325939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.325939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.326317 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:31.326469 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.326802 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.326989 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260925 07:23:31.328291 19699 catalog_manager.cc:1347] Generated new cluster ID: f9dc68eaf56e4485a474ada50305c4cb
I20260925 07:23:31.328343 19699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:23:31.363363 19699 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:23:31.364197 19699 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:31.372967 19699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Generated new TSK 0
I20260925 07:23:31.373201 19699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:23:31.405961 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:31.488391 19713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:31.488647 19713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:31.488685 19713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:31.489847 19713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:31.489917 19713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:31.489960 19713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:31.491466 19713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:31.491731 19713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:31.491961 19713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:31.497058 19718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.497129 19713 server_base.cc:1034] running on GCE node
W20260925 07:23:31.497053 19721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.497053 19719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.497490 19713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:31.497746 19713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:31.498917 19713 hybrid_clock.cc:648] HybridClock initialized: now 1790321011498910 us; error 41 us; skew 500 ppm
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:31.501554 19713 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:31.502066 19713 webserver.cc:460] Webserver started at http://127.4.47.1:32885/ using document root <none> and password file <none>
I20260925 07:23:31.502298 19713 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:31.502377 19713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:31.502533 19713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:31.503690 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "7e334b12b3b468e35e50ccbf9a3d1e1d"
server_key_iv: "7127be4ff6593a9c427484e79015e2a0"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.504048 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "7e334b12b3b468e35e50ccbf9a3d1e1d"
server_key_iv: "7127be4ff6593a9c427484e79015e2a0"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.505487 19713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:23:31.506219 19728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.506394 19713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:31.506528 19713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "7e334b12b3b468e35e50ccbf9a3d1e1d"
server_key_iv: "7127be4ff6593a9c427484e79015e2a0"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.506606 19713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:31.518427 19713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:31.520593 19713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:31.520758 19713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:31.520983 19713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:31.521350 19713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:31.521407 19713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.521456 19713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:31.521484 19713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.527588 19713 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:40637
I20260925 07:23:31.527642 19841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:40637 every 8 connection(s)
I20260925 07:23:31.527962 19713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:31.531414 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19713
I20260925 07:23:31.531546 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:23:31.531754 4284 external_mini_cluster.cc:1428] Setting key 54196138999e42c9747ae695b0173437
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:31.538517 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.528678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:46985)
0925 07:23:31.528847 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.528888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.529363 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.529575 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.529578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.530144 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:31.530337 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.530342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.531016 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.531016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.532093 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.532096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.532634 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.532636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.532784 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.533281 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:31.533292 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:31.535250 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:31.537132 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.537138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.537140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.537357 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.537584 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.537585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.537585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.537686 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.538004 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:31.538009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:31.538130 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.538218 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.538342 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 07:23:31.538506 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.528721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:46985 (local address 127.4.47.62:33747)
0925 07:23:31.528765 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.528767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.528988 (+ 221us) server_negotiation.cc:407] Connection header received
0925 07:23:31.529447 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.529448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.529460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.529522 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:31.530417 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.530919 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.532179 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.532511 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.532591 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.535354 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:31.535355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:31.535356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:31.535364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:31.537044 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.537438 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.537440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.537440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.537509 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.537753 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.537754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.537755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.537919 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:31.538010 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.538269 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.538370 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:23:31.539109 19842 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:31.539230 19842 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:31.539428 19842 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:31.540040 19651 ts_manager.cc:194] Registered new tserver with Master: b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637)
I20260925 07:23:31.540711 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:46985
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:23:31.562274 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:31.644240 19847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:31.644455 19847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:31.644480 19847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:31.645560 19847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:31.645593 19847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:31.645619 19847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:31.647112 19847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:31.647306 19847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:31.647491 19847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:31.652561 19853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.652567 19852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.652559 19855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.652868 19847 server_base.cc:1034] running on GCE node
I20260925 07:23:31.653084 19847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:31.653339 19847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:31.654507 19847 hybrid_clock.cc:648] HybridClock initialized: now 1790321011654511 us; error 51 us; skew 500 ppm
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:31.657155 19847 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:31.657644 19847 webserver.cc:460] Webserver started at http://127.4.47.2:42071/ using document root <none> and password file <none>
I20260925 07:23:31.657891 19847 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:31.657936 19847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:31.658100 19847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:31.659245 19847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "81d802ab230840f3b62b2af571c25fb4"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "2e01298e09d6b84696607d786d3342b5"
server_key_iv: "a918d0319b5c650466d628c1b3e476de"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.659601 19847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "81d802ab230840f3b62b2af571c25fb4"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "2e01298e09d6b84696607d786d3342b5"
server_key_iv: "a918d0319b5c650466d628c1b3e476de"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.660905 19847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:31.661657 19862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.661829 19847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:31.661912 19847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "81d802ab230840f3b62b2af571c25fb4"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "2e01298e09d6b84696607d786d3342b5"
server_key_iv: "a918d0319b5c650466d628c1b3e476de"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.661973 19847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:31.700757 19847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:31.703261 19847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:31.703384 19847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:31.703573 19847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:31.703910 19847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:31.703972 19847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.704034 19847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:31.704069 19847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.710131 19847 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:41411
I20260925 07:23:31.710175 19975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:41411 every 8 connection(s)
I20260925 07:23:31.710578 19847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:31.717730 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19847
I20260925 07:23:31.717868 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:23:31.718109 4284 external_mini_cluster.cc:1428] Setting key 042b03a423fc926cbc4a57524719689f
I20260925 07:23:31.720625 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.711400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:56061 (local address 127.4.47.62:33747)
0925 07:23:31.711448 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.711451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.711693 (+ 242us) server_negotiation.cc:407] Connection header received
0925 07:23:31.712169 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.712170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.712181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.712230 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:31.713138 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.713632 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.714658 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.714879 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.714985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.717310 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:31.717312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:31.717312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:31.717319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:31.719222 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.719614 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.719615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.719616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.719685 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.719923 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.719924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.719924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.720084 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:31.720157 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.720453 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.720535 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:23:31.720842 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.711359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:56061)
0925 07:23:31.711565 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.711602 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.712081 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.712291 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.712294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.712853 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:31.713055 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.713060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.713717 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.713718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.714549 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.714551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.715047 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.715047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.715224 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.715719 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:31.715727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:31.717220 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:31.719302 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.719308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.719310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.719534 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.719756 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.719756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.719757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.719855 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.720151 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:31.720155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:31.720310 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.720422 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.720587 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":140,"threads_started":1}
I20260925 07:23:31.721283 19976 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:31.721403 19976 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:31.721627 19976 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:31.722007 19651 ts_manager.cc:194] Registered new tserver with Master: 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
I20260925 07:23:31.722398 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:56061
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:23:31.746577 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:31.835160 19981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:31.835420 19981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:31.835462 19981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:31.836687 19981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:31.836758 19981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:31.836805 19981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:31.838313 19981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:31.838593 19981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:31.838827 19981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:31.843969 19989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.843966 19986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:31.843966 19987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:31.844252 19981 server_base.cc:1034] running on GCE node
I20260925 07:23:31.844457 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:31.844707 19981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:31.845878 19981 hybrid_clock.cc:648] HybridClock initialized: now 1790321011845856 us; error 38 us; skew 500 ppm
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:31.848608 19981 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:31.849072 19981 webserver.cc:460] Webserver started at http://127.4.47.3:39355/ using document root <none> and password file <none>
I20260925 07:23:31.849304 19981 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:31.849375 19981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:31.849534 19981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:23:31.850590 19981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "23f9b20eb5b54c56a38da2e547f091ab"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "d88c5d4771c7758a20d76818a0fece14"
server_key_iv: "91011bb32b6d2b9d483243d90498fa33"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.850955 19981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "23f9b20eb5b54c56a38da2e547f091ab"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "d88c5d4771c7758a20d76818a0fece14"
server_key_iv: "91011bb32b6d2b9d483243d90498fa33"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.852372 19981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:31.853171 19996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.853349 19981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:23:31.853448 19981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "23f9b20eb5b54c56a38da2e547f091ab"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "d88c5d4771c7758a20d76818a0fece14"
server_key_iv: "91011bb32b6d2b9d483243d90498fa33"
server_key_version: "encryptionkey@0"
I20260925 07:23:31.853566 19981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:31.866279 19981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:31.868255 19981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:31.868425 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:31.868671 19981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:31.869014 19981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:31.869068 19981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.869118 19981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:31.869146 19981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:31.875597 19981 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:43483
I20260925 07:23:31.875636 20109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:43483 every 8 connection(s)
I20260925 07:23:31.875964 19981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:31.881790 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 19981
I20260925 07:23:31.881879 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:23:31.882071 4284 external_mini_cluster.cc:1428] Setting key f2a6776d5bed5fa00afd42328ad4e43e
Sep 25 07:23:31 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:31.885268 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.876798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:58329 (local address 127.4.47.62:33747)
0925 07:23:31.876856 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.876858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.876971 (+ 113us) server_negotiation.cc:407] Connection header received
0925 07:23:31.877432 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.877433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.877462 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.877524 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:31.878468 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.878970 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.880028 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.880215 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.880272 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.882625 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:31.882626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:31.882626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:31.882633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:31.884078 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.884463 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.884464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.884464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.884515 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.884734 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.884734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.884735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.884834 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:31.884884 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.885141 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.885204 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:23:31.885427 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.876679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:58329)
0925 07:23:31.876838 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.876880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.877349 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.877577 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.877580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.878154 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:31.878348 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.878353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.879061 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.879061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.879947 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.879949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.880351 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.880352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.880515 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.880997 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:31.881006 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:31.882515 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:31.884164 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.884170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.884171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.884387 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.884574 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:31.884575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:31.884575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:31.884678 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:31.884890 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:31.884894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:31.885018 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.885102 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.885223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 07:23:31.885933 20110 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:31.886050 20110 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:31.886276 20110 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:31.886711 19651 ts_manager.cc:194] Registered new tserver with Master: 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483)
I20260925 07:23:31.887154 19651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:58329
I20260925 07:23:31.893916 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:31.901736 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.895361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.4.47.62:33747)
0925 07:23:31.895434 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.895437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.895504 (+ 67us) server_negotiation.cc:407] Connection header received
0925 07:23:31.895606 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.895607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.895616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.895699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:31.896268 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.896776 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.897513 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.897721 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.897797 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.899008 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:31.899009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:31.899009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:31.899017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:31.900471 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.900886 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.900887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.900887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.900951 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:31.901152 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:31.901153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:31.901153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:31.901312 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:31.901429 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.901551 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.901631 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:23:31.903578 19651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:23:31.905197 19651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:23:31.911603 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.907573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43483 (local address 127.0.0.1:51634)
0925 07:23:31.907746 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.907786 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.907851 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.908112 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.908114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.908116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:31.908190 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.908193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909433 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.909434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.910584 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.910587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.911358 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.911359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.911415 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.911426 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.911478 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.911537 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260925 07:23:31.911635 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.907763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.4.47.3:43483)
0925 07:23:31.907951 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.907956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.907968 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:31.908010 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.908013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.908030 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.908111 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:31.908347 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909323 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.910666 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.911283 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.911361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.911466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.911494 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.911533 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260925 07:23:31.912334 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.907611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41411 (local address 127.0.0.1:37374)
0925 07:23:31.907930 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.907941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.908005 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.908508 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.908508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.908511 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:31.908552 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.908553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909906 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.909907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.911151 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.911154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.912097 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.912098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.912148 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.912151 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.912227 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.912271 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":64,"thread_start_us":90,"threads_started":1}
I20260925 07:23:31.912375 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.907753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.4.47.2:41411)
0925 07:23:31.908241 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.908246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.908255 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:31.908294 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.908296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.908311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.908385 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:31.908621 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909788 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.911253 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.912015 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.912097 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.912149 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.912246 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.912292 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":83,"threads_started":1}
I20260925 07:23:31.912438 20044 tablet_service.cc:1467] Processing CreateTablet for tablet 7500342108db43cdab335198f24957c1 (DEFAULT_TABLE table=test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:31.912859 20044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7500342108db43cdab335198f24957c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:31.913194 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.907799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.0.0.1:60116)
0925 07:23:31.908020 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:31.908031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:31.908080 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:31.908456 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:31.908457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:31.908459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:31.908512 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.908514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909906 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.909907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.911152 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:31.911154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.912959 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.912960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:31.913024 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.913029 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:23:31.913093 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:23:31.913131 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":57,"thread_start_us":72,"threads_started":1}
I20260925 07:23:31.913234 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:31.908036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.4.47.1:40637)
0925 07:23:31.908214 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:31.908217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:31.908225 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:23:31.908284 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:31.908286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:31.908302 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:31.908374 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:31.908592 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.909770 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.912186 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:31.912880 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:31.912959 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:31.913015 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:31.913107 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 07:23:31.913145 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260925 07:23:31.913281 19910 tablet_service.cc:1467] Processing CreateTablet for tablet 7500342108db43cdab335198f24957c1 (DEFAULT_TABLE table=test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:31.913568 19910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7500342108db43cdab335198f24957c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:31.913823 19776 tablet_service.cc:1467] Processing CreateTablet for tablet 7500342108db43cdab335198f24957c1 (DEFAULT_TABLE table=test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:31.914120 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7500342108db43cdab335198f24957c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:31.916185 20127 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Bootstrap starting.
I20260925 07:23:31.916816 20127 tablet_bootstrap.cc:654] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:31.916783 20128 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Bootstrap starting.
I20260925 07:23:31.917389 20127 log.cc:824] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:31.917491 20128 tablet_bootstrap.cc:654] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:31.917445 20129 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap starting.
I20260925 07:23:31.918087 20128 log.cc:824] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:31.918121 20129 tablet_bootstrap.cc:654] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:31.918267 20127 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: No bootstrap required, opened a new log
I20260925 07:23:31.918349 20127 ts_tablet_manager.cc:1397] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:31.918685 20129 log.cc:824] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:31.919473 20129 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: No bootstrap required, opened a new log
I20260925 07:23:31.919605 20129 ts_tablet_manager.cc:1397] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:31.919629 20128 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: No bootstrap required, opened a new log
I20260925 07:23:31.919687 20128 ts_tablet_manager.cc:1397] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:23:31.920511 20127 raft_consensus.cc:348] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.920697 20127 raft_consensus.cc:374] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:31.920734 20127 raft_consensus.cc:729] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d802ab230840f3b62b2af571c25fb4, State: Initialized, Role: FOLLOWER
I20260925 07:23:31.920882 20127 consensus_queue.cc:260] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.921159 20127 ts_tablet_manager.cc:1428] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:23:31.921139 20129 raft_consensus.cc:348] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.921283 19976 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
I20260925 07:23:31.921308 20129 raft_consensus.cc:374] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:31.921370 20129 raft_consensus.cc:729] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d35f784f4d0d981ded1fb9d44257, State: Initialized, Role: FOLLOWER
I20260925 07:23:31.921497 20129 consensus_queue.cc:260] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.921775 20129 ts_tablet_manager.cc:1428] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:31.921695 20128 raft_consensus.cc:348] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.921798 19842 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
I20260925 07:23:31.921865 20128 raft_consensus.cc:374] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:31.921924 20128 raft_consensus.cc:729] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f9b20eb5b54c56a38da2e547f091ab, State: Initialized, Role: FOLLOWER
I20260925 07:23:31.922065 20128 consensus_queue.cc:260] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:31.922394 20110 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
I20260925 07:23:31.922394 20128 ts_tablet_manager.cc:1428] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 07:23:31.961170 19977 tablet.cc:2369] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:32.026738 20135 raft_consensus.cc:482] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:32.026961 20135 raft_consensus.cc:504] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:32.027460 20135 leader_election.cc:288] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:23:32.028697 19843 tablet.cc:2369] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:32.031093 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.027720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:52279 (local address 127.4.47.1:40637)
0925 07:23:32.027762 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:32.027765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:32.027933 (+ 168us) server_negotiation.cc:407] Connection header received
0925 07:23:32.028014 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:32.028015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:32.028028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:32.028089 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:32.028272 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.028720 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.030320 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.030726 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.030787 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:32.030795 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:32.030956 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:23:32.030999 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:23:32.031123 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.027637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.4.47.3:52279)
0925 07:23:32.027863 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:32.027874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:32.027938 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:32.028102 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:32.028120 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:32.028122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:32.028193 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:32.028195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.028812 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.028813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:32.030219 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:32.030222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.030814 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.030815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:32.030869 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:32.030873 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:32.030947 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:23:32.030998 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":75,"thread_start_us":116,"threads_started":1}
I20260925 07:23:32.031512 19796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500342108db43cdab335198f24957c1" candidate_uuid: "23f9b20eb5b54c56a38da2e547f091ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" is_pre_election: true
I20260925 07:23:32.031699 19796 raft_consensus.cc:2454] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23f9b20eb5b54c56a38da2e547f091ab in term 0.
I20260925 07:23:32.031756 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.027637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41411 (local address 127.4.47.3:42117)
0925 07:23:32.027703 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:32.027716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:32.027759 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:32.027987 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:32.027988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:32.027990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:32.028070 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:32.028073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.028872 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.028873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:32.030219 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:32.030222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.031486 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.031487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:32.031576 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:32.031579 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:23:32.031644 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:23:32.031684 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":121}
I20260925 07:23:32.031796 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.027716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:42117 (local address 127.4.47.2:41411)
0925 07:23:32.027763 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:32.027765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:32.027803 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:23:32.027836 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:32.027838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:32.027851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:32.027928 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:32.028150 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.028793 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.030834 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.031383 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.031460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:32.031467 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:32.031674 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 07:23:32.031718 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:23:32.032004 19997 leader_election.cc:302] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f9b20eb5b54c56a38da2e547f091ab, b8c2d35f784f4d0d981ded1fb9d44257; no voters:
I20260925 07:23:32.032130 20135 raft_consensus.cc:2781] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:32.032183 20135 raft_consensus.cc:482] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:32.032212 20135 raft_consensus.cc:3037] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:32.032217 19930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500342108db43cdab335198f24957c1" candidate_uuid: "23f9b20eb5b54c56a38da2e547f091ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d802ab230840f3b62b2af571c25fb4" is_pre_election: true
I20260925 07:23:32.032366 19930 raft_consensus.cc:2454] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23f9b20eb5b54c56a38da2e547f091ab in term 0.
I20260925 07:23:32.033097 20135 raft_consensus.cc:504] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:32.033241 20135 leader_election.cc:288] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [CANDIDATE]: Term 1 election: Requested vote from peers b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
I20260925 07:23:32.033391 19796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500342108db43cdab335198f24957c1" candidate_uuid: "23f9b20eb5b54c56a38da2e547f091ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
I20260925 07:23:32.033479 19796 raft_consensus.cc:3037] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:32.033471 19930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7500342108db43cdab335198f24957c1" candidate_uuid: "23f9b20eb5b54c56a38da2e547f091ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d802ab230840f3b62b2af571c25fb4"
I20260925 07:23:32.033569 19930 raft_consensus.cc:3037] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:32.034278 19796 raft_consensus.cc:2454] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23f9b20eb5b54c56a38da2e547f091ab in term 1.
I20260925 07:23:32.034289 19930 raft_consensus.cc:2454] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23f9b20eb5b54c56a38da2e547f091ab in term 1.
I20260925 07:23:32.034471 20000 leader_election.cc:302] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4; no voters:
I20260925 07:23:32.034562 20135 raft_consensus.cc:2781] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:32.034708 20135 raft_consensus.cc:686] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 LEADER]: Becoming Leader. State: Replica: 23f9b20eb5b54c56a38da2e547f091ab, State: Running, Role: LEADER
I20260925 07:23:32.034804 20135 consensus_queue.cc:237] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:32.035781 19651 catalog_manager.cc:5495] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3). New cstate: current_term: 1 leader_uuid: "23f9b20eb5b54c56a38da2e547f091ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20260925 07:23:32.061211 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.058568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.4.47.3:43483)
0925 07:23:32.058650 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:32.058651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:32.058659 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:23:32.058696 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:32.058697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:32.058717 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:32.058794 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:32.058946 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.059494 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.060268 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:32.060506 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:32.060605 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:32.060810 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:32.060916 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:32.061024 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:32.061044 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 07:23:32.061068 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:23:32.063158 19796 raft_consensus.cc:1261] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Refusing update from remote peer 23f9b20eb5b54c56a38da2e547f091ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:32.063170 19930 raft_consensus.cc:1261] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Refusing update from remote peer 23f9b20eb5b54c56a38da2e547f091ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:32.063546 20135 consensus_queue.cc:1035] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:32.063719 20137 consensus_queue.cc:1035] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:32.065816 20140 mvcc.cc:205] Tried to move back new op lower bound from 7333154865407840256 to 7333154865295269888. Current Snapshot: MvccSnapshot[applied={T|T < 7333154865407840256}]
I20260925 07:23:32.066201 20142 mvcc.cc:205] Tried to move back new op lower bound from 7333154865407840256 to 7333154865295269888. Current Snapshot: MvccSnapshot[applied={T|T < 7333154865407840256}]
I20260925 07:23:32.066470 20141 mvcc.cc:205] Tried to move back new op lower bound from 7333154865407840256 to 7333154865295269888. Current Snapshot: MvccSnapshot[applied={T|T < 7333154865407840256}]
I20260925 07:23:32.068742 19651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:32.068848 19651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:32.069618 19651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Sending DeleteTablet for 3 replicas of tablet 7500342108db43cdab335198f24957c1
I20260925 07:23:32.069962 19910 tablet_service.cc:1514] Processing DeleteTablet for tablet 7500342108db43cdab335198f24957c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:32 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:37374
I20260925 07:23:32.070005 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19713
I20260925 07:23:32.070044 20044 tablet_service.cc:1514] Processing DeleteTablet for tablet 7500342108db43cdab335198f24957c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:32 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:51634
I20260925 07:23:32.070600 20151 tablet_replica.cc:330] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: stopping tablet replica
I20260925 07:23:32.070693 20151 raft_consensus.cc:2229] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:23:32.070814 20151 raft_consensus.cc:2258] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:32.070847 20150 tablet_replica.cc:330] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: stopping tablet replica
I20260925 07:23:32.071048 20150 raft_consensus.cc:2229] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:32.071228 20150 raft_consensus.cc:2258] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:32.071393 20151 ts_tablet_manager.cc:1905] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:32.071755 20150 ts_tablet_manager.cc:1905] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:32.073304 20151 ts_tablet_manager.cc:1918] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:32.073406 20151 log.cc:1196] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/7500342108db43cdab335198f24957c1
I20260925 07:23:32.073485 20150 ts_tablet_manager.cc:1918] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:32.073586 20150 log.cc:1196] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/7500342108db43cdab335198f24957c1
I20260925 07:23:32.073748 20151 ts_tablet_manager.cc:1939] T 7500342108db43cdab335198f24957c1 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting consensus metadata
I20260925 07:23:32.073870 20150 ts_tablet_manager.cc:1939] T 7500342108db43cdab335198f24957c1 P 81d802ab230840f3b62b2af571c25fb4: Deleting consensus metadata
I20260925 07:23:32.074206 19635 catalog_manager.cc:4841] TS 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): tablet 7500342108db43cdab335198f24957c1 (table test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]) successfully deleted
I20260925 07:23:32.074220 19638 catalog_manager.cc:4841] TS 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): tablet 7500342108db43cdab335198f24957c1 (table test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]) successfully deleted
W20260925 07:23:32.074506 19635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:40637 (error 108)
I20260925 07:23:32.074991 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:32.074857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.0.0.1:60132)
0925 07:23:32.074915 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:32.074952 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 07:23:32.075130 19635 catalog_manager.cc:4584] TS b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7500342108db43cdab335198f24957c1: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40637: connect: Connection refused (error 111)
I20260925 07:23:32.075623 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19847
I20260925 07:23:32.082378 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19981
I20260925 07:23:32.087213 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 19619
I20260925 07:23:52.093048 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--webserver_interface=127.4.47.62
--webserver_port=39885
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:23:52.175040 20153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:52.175297 20153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:52.175320 20153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:52.176503 20153 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:23:52.176549 20153 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:23:52.176563 20153 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:23:52.176574 20153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:23:52.176587 20153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:52.176600 20153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:52.176651 20153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:23:52.176668 20153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:23:52.178293 20153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:52.178552 20153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:52.178786 20153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:52.187681 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.187924 20153 server_base.cc:1034] running on GCE node
W20260925 07:23:52.187687 20159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:52.187675 20160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.188459 20153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:52.188809 20153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:52.189962 20153 hybrid_clock.cc:648] HybridClock initialized: now 1790321032189950 us; error 37 us; skew 500 ppm
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:52.192760 20153 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:23:52.193253 20153 webserver.cc:460] Webserver started at http://127.4.47.62:39885/ using document root <none> and password file <none>
I20260925 07:23:52.193526 20153 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:52.193617 20153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:52.195264 20153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:23:52.195999 20169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.196269 20153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.196394 20153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "e0628f20f5374a798d577a13495caa7d"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "6b17412672b7e4c7e378d2fe8e7920f1"
server_key_iv: "027f0ad2047ac01098bd487eb34586fe"
server_key_version: "encryptionkey@0"
I20260925 07:23:52.196827 20153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:52.222453 20153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:52.224625 20153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:52.224792 20153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:52.229059 20153 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33747
I20260925 07:23:52.229100 20221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33747 every 8 connection(s)
I20260925 07:23:52.229542 20153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:23:52.231127 20222 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:23:52.231830 20222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap starting.
I20260925 07:23:52.239130 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20153
I20260925 07:23:52.239483 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:40637
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=32885
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:52.240679 20222 log.cc:824] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:52.242832 20222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:52.243254 20222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap complete.
I20260925 07:23:52.245661 20222 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:52.245910 20222 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Initialized, Role: FOLLOWER
I20260925 07:23:52.246089 20222 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:52.246150 20222 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:23:52.246212 20222 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:23:52.246270 20222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:23:52.247390 20222 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:52.247478 20222 leader_election.cc:302] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0628f20f5374a798d577a13495caa7d; no voters:
I20260925 07:23:52.247715 20222 leader_election.cc:288] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:23:52.247759 20226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:23:52.247887 20226 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 LEADER]: Becoming Leader. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Running, Role: LEADER
I20260925 07:23:52.247987 20222 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:23:52.247972 20226 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:23:52.248247 20226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0628f20f5374a798d577a13495caa7d. Latest consensus state: current_term: 2 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:23:52.248312 20226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:52.248312 20228 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:23:52.248456 20228 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:23:52.248566 20231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:23:52.249435 20231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:23:52.250027 20231 catalog_manager.cc:1259] Loaded cluster ID: f9dc68eaf56e4485a474ada50305c4cb
I20260925 07:23:52.250072 20231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:23:52.251361 20231 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:23:52.251554 20231 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Loaded TSK: 0
I20260925 07:23:52.260730 20231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Generated new TSK 1
I20260925 07:23:52.260833 20231 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:23:52.327664 20224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:52.327879 20224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:52.327904 20224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:52.329072 20224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:52.329114 20224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:52.329142 20224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:23:52.330582 20224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:40637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=32885
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:52.330822 20224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:52.331110 20224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:52.337407 20249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:52.337419 20250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.337586 20224 server_base.cc:1034] running on GCE node
W20260925 07:23:52.337419 20252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.338006 20224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:52.338284 20224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:52.339443 20224 hybrid_clock.cc:648] HybridClock initialized: now 1790321032339433 us; error 31 us; skew 500 ppm
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:52.341961 20224 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:23:52.342527 20224 webserver.cc:460] Webserver started at http://127.4.47.1:32885/ using document root <none> and password file <none>
I20260925 07:23:52.342767 20224 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:52.342825 20224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:52.344302 20224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.345055 20259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.345252 20224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.345350 20224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "7e334b12b3b468e35e50ccbf9a3d1e1d"
server_key_iv: "7127be4ff6593a9c427484e79015e2a0"
server_key_version: "encryptionkey@0"
I20260925 07:23:52.345722 20224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:52.368122 20224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:52.370297 20224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:52.370481 20224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:52.370724 20224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:52.371188 20266 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:23:52.373313 20224 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:23:52.373376 20224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:23:52.373431 20224 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:23:52.374084 20224 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:23:52.374135 20224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:23:52.374207 20266 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap starting.
I20260925 07:23:52.381239 20224 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:40637
I20260925 07:23:52.381322 20373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:40637 every 8 connection(s)
I20260925 07:23:52.381635 20224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:23:52.384416 20266 log.cc:824] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:52.385187 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20224
I20260925 07:23:52.385514 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:41411
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=42071
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:23:52.386951 20266 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:23:52.387279 20266 tablet_bootstrap.cc:492] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap complete.
I20260925 07:23:52.388427 20266 ts_tablet_manager.cc:1397] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:52.389624 20266 raft_consensus.cc:348] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:52.389933 20266 raft_consensus.cc:729] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d35f784f4d0d981ded1fb9d44257, State: Initialized, Role: FOLLOWER
I20260925 07:23:52.390076 20266 consensus_queue.cc:260] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } }
I20260925 07:23:52.390381 20266 ts_tablet_manager.cc:1428] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:23:52.393719 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.382338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:37039)
0925 07:23:52.382531 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.382575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.383133 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.383611 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.383615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.383961 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:52.384150 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.384155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.385530 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.385532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.386778 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.386781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.387711 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.387712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.387880 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.388425 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:52.388435 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:52.390019 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:52.392080 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.392088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.392090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.392316 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.392614 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.392615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.392615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.392722 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.393090 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:52.393097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:52.393236 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.393342 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.393523 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260925 07:23:52.393719 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.382460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:37039 (local address 127.4.47.62:33747)
0925 07:23:52.382668 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.382676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.382719 (+ 43us) server_negotiation.cc:407] Connection header received
0925 07:23:52.383392 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.383398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.383443 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.383583 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.384264 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.385371 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.386900 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.387595 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.387721 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.390135 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.390139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.390139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.390157 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.391997 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.392403 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.392406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.392407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.392510 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.392789 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.392790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.392791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.393019 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.393112 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.393402 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.393532 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260925 07:23:52.394546 20374 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:52.394658 20374 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:52.394878 20374 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:52.395743 20183 ts_manager.cc:194] Registered new tserver with Master: b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637)
I20260925 07:23:52.396842 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:37039
I20260925 07:23:52.398011 20374 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.1@KRBTEST.COM
I20260925 07:23:52.404680 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.396479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.4.47.1:40637)
0925 07:23:52.396672 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.396677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.396691 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:23:52.396800 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.396802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.396819 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.396896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.397501 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.398296 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.399154 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.399493 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.399582 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.401743 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.401750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.401750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.401756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.403418 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.403879 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.403882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.403882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.403937 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.404126 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.404126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.404127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.404275 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.404339 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.404520 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.404599 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260925 07:23:52.404780 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.396406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.0.0.1:53024)
0925 07:23:52.396600 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.396637 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.396704 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.396906 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.396910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.397347 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:52.397420 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.397423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.398377 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.398378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.399020 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.399022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.399566 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.399569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.399651 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.400161 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:52.400166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:52.401657 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:52.403493 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.403500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.403502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.403783 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.403995 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.403996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.403996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.404063 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.404334 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:52.404342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:52.404402 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.404495 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.404628 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
I20260925 07:23:52.405242 20308 tablet_service.cc:1514] Processing DeleteTablet for tablet 7500342108db43cdab335198f24957c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:32 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:53024
I20260925 07:23:52.405866 20383 tablet_replica.cc:330] stopping tablet replica
I20260925 07:23:52.405985 20383 raft_consensus.cc:2229] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:52.406073 20383 raft_consensus.cc:2258] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:52.406364 20383 ts_tablet_manager.cc:1905] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:52.408625 20383 ts_tablet_manager.cc:1918] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:52.408718 20383 log.cc:1196] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal/wals/7500342108db43cdab335198f24957c1
I20260925 07:23:52.408927 20383 ts_tablet_manager.cc:1939] T 7500342108db43cdab335198f24957c1 P b8c2d35f784f4d0d981ded1fb9d44257: Deleting consensus metadata
I20260925 07:23:52.409467 20170 catalog_manager.cc:4841] TS b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637): tablet 7500342108db43cdab335198f24957c1 (table test-table [id=35746c2c0d694a0fa7f94d22a8b9a5c3]) successfully deleted
W20260925 07:23:52.473631 20379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:52.473834 20379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:52.473856 20379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:52.474936 20379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:52.474970 20379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:52.474995 20379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:23:52.476410 20379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:41411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=42071
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:52.476585 20379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:52.476861 20379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:52.482149 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:52.482216 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:52.482147 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.482184 20379 server_base.cc:1034] running on GCE node
I20260925 07:23:52.482563 20379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:52.482758 20379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:52.483916 20379 hybrid_clock.cc:648] HybridClock initialized: now 1790321032483903 us; error 41 us; skew 500 ppm
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:52.486342 20379 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:23:52.486774 20379 webserver.cc:460] Webserver started at http://127.4.47.2:42071/ using document root <none> and password file <none>
I20260925 07:23:52.487015 20379 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:52.487107 20379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:52.488533 20379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.489068 20398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.489234 20379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:23:52.489331 20379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "81d802ab230840f3b62b2af571c25fb4"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "2e01298e09d6b84696607d786d3342b5"
server_key_iv: "a918d0319b5c650466d628c1b3e476de"
server_key_version: "encryptionkey@0"
I20260925 07:23:52.489676 20379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:52.500433 20379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:52.502419 20379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:52.502571 20379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:52.502792 20379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:52.503132 20379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:52.503175 20379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.503214 20379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:52.503242 20379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.509194 20379 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:41411
I20260925 07:23:52.509236 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:41411 every 8 connection(s)
I20260925 07:23:52.509562 20379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:23:52.513397 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20379
I20260925 07:23:52.513697 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:43483
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=39355
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:52.519243 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.510225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:36959 (local address 127.4.47.62:33747)
0925 07:23:52.510288 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.510290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.510513 (+ 223us) server_negotiation.cc:407] Connection header received
0925 07:23:52.511015 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.511016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.511029 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.511151 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.512054 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.512563 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.513599 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.513926 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.513998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.516298 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.516299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.516299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.516306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.517742 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.518244 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.518245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.518245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.518292 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.518577 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.518578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.518578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.518676 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.518725 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.519098 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.519179 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:23:52.519575 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.510230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:36959)
0925 07:23:52.510412 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.510448 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.510931 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.511217 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.511221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.511779 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:52.511977 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.511981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.512640 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.512641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.513526 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.513529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.514004 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.514005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.514113 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.514633 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:52.514641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:52.516198 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:52.517824 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.517832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.517835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.518167 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.518359 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.518360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.518360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.518512 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.518738 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:52.518744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:52.518924 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.519051 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.519275 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260925 07:23:52.520112 20512 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:52.520201 20512 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:52.520391 20512 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:52.520776 20183 ts_manager.cc:194] Registered new tserver with Master: 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
I20260925 07:23:52.521142 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:36959
W20260925 07:23:52.596266 20515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:23:52.596510 20515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:23:52.596552 20515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:23:52.597744 20515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:23:52.597812 20515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:23:52.597872 20515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:23:52.599907 20515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:43483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=39355
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:23:52.600234 20515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:23:52.600471 20515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:23:52.605820 20520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.605846 20515 server_base.cc:1034] running on GCE node
W20260925 07:23:52.605827 20521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:23:52.605880 20523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:23:52.606251 20515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:23:52.606508 20515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:23:52.607663 20515 hybrid_clock.cc:648] HybridClock initialized: now 1790321032607650 us; error 27 us; skew 500 ppm
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:23:52.610256 20515 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:23:52.610770 20515 webserver.cc:460] Webserver started at http://127.4.47.3:39355/ using document root <none> and password file <none>
I20260925 07:23:52.611044 20515 fs_manager.cc:359] Metadata directory not provided
I20260925 07:23:52.611181 20515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:23:52.612646 20515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.613314 20530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.613511 20515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:23:52.613611 20515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "23f9b20eb5b54c56a38da2e547f091ab"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "d88c5d4771c7758a20d76818a0fece14"
server_key_iv: "91011bb32b6d2b9d483243d90498fa33"
server_key_version: "encryptionkey@0"
I20260925 07:23:52.613996 20515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:23:52.639385 20515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:23:52.641388 20515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:23:52.641558 20515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:23:52.641793 20515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:23:52.642107 20515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:23:52.642164 20515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.642204 20515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:23:52.642232 20515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.648286 20515 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:43483
I20260925 07:23:52.648331 20643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:43483 every 8 connection(s)
I20260925 07:23:52.648746 20515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:23:52.649242 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20515
Sep 25 07:23:52 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:23:52.658238 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.649619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.4.47.62:33747)
0925 07:23:52.649747 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.649749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.650627 (+ 878us) server_negotiation.cc:407] Connection header received
0925 07:23:52.650687 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.650687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.650695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.650741 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.651579 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.652333 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.653453 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.653651 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.653707 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.655252 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.655253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.655254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.655260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.656975 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.657385 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.657386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.657386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.657463 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.657684 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.657685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.657685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.657793 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.657924 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.658032 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.658130 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260925 07:23:52.659699 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.649585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:43165 (local address 127.4.47.62:33747)
0925 07:23:52.649725 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.649727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.649959 (+ 232us) server_negotiation.cc:407] Connection header received
0925 07:23:52.650433 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.650434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.650444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.650499 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.651834 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.652482 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.653813 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.654009 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.654072 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.656686 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.656687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.656688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.656694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.658275 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.658796 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.658797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.658797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.658870 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.659175 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.659176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.659176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.659270 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.659323 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.659581 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.659641 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 07:23:52.659933 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.649600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:43165)
0925 07:23:52.649813 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.649867 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.650351 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.650624 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.650628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.651482 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:23:52.651740 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.651745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.652562 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.652563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.653737 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.653739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.654088 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.654089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.654218 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.654686 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:23:52.654696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:23:52.656593 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:23:52.658437 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.658444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.658446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.658726 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.658942 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:23:52.658943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:23:52.658943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:23:52.659117 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:23:52.659315 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:23:52.659318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:23:52.659467 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.659545 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.659667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260925 07:23:52.660390 20644 heartbeater.cc:347] Connected to a master server at 127.4.47.62:33747
I20260925 07:23:52.660488 20644 heartbeater.cc:464] Registering TS with master...
I20260925 07:23:52.660661 20644 heartbeater.cc:511] Master 127.4.47.62:33747 requested a full tablet report, sending...
I20260925 07:23:52.661041 20183 ts_manager.cc:194] Registered new tserver with Master: 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483)
I20260925 07:23:52.661460 20183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:43165
I20260925 07:23:52.670047 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:23:52.672770 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.670477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.4.47.62:33747)
0925 07:23:52.670542 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.670544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.670617 (+ 73us) server_negotiation.cc:407] Connection header received
0925 07:23:52.670685 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.670685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.670721 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.670784 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:52.670963 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.671479 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.672217 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.672391 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.672440 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.672557 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:52.672603 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 07:23:52.672720 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40124: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 07:23:52.672859 20377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40124: token expired
W20260925 07:23:52.672957 20113 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.47.62:33747)
I20260925 07:23:52.679039 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.673211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.4.47.62:33747)
0925 07:23:52.673247 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.673249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.673303 (+ 54us) server_negotiation.cc:407] Connection header received
0925 07:23:52.673387 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.673388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.673397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.673447 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:23:52.673952 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.674403 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.675134 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.675332 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.675404 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.676617 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:23:52.676619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:23:52.676619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:23:52.676627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:23:52.677900 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.678250 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.678251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.678252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.678346 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:23:52.678544 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:23:52.678545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:23:52.678545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:23:52.678654 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:23:52.678761 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.678858 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.678934 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:23:52.680169 20183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:23:52.681762 20183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:23:52.683926 20308 tablet_service.cc:1467] Processing CreateTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 (DEFAULT_TABLE table=test-table [id=cdd2bc37739844eb94f0d6c94f939fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:52.684167 20308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213f34ffa96d44f5afd86976e7cccbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:52.685783 20266 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap starting.
I20260925 07:23:52.686110 20266 tablet_bootstrap.cc:654] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:52.686945 20266 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: No bootstrap required, opened a new log
I20260925 07:23:52.687013 20266 ts_tablet_manager.cc:1397] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:23:52.687188 20266 raft_consensus.cc:348] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.687266 20266 raft_consensus.cc:374] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:52.687331 20266 raft_consensus.cc:729] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d35f784f4d0d981ded1fb9d44257, State: Initialized, Role: FOLLOWER
I20260925 07:23:52.687386 20266 consensus_queue.cc:260] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.687493 20266 ts_tablet_manager.cc:1428] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:23:52.687747 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.683555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43483 (local address 127.0.0.1:43834)
0925 07:23:52.683608 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.683621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.683659 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.684259 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.684260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.684262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:52.684310 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.684312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.685214 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.685215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.686321 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.686323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.687444 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.687445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.687532 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.687559 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.687612 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.687669 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 07:23:52.687747 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.683601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41411 (local address 127.0.0.1:35230)
0925 07:23:52.683783 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.683793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.683830 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.684272 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.684274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.684276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:52.684340 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.684342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.685615 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.685616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.686652 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.686654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.687559 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.687559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.687626 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.687628 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.687657 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.687684 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
I20260925 07:23:52.688033 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.683881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.4.47.2:41411)
0925 07:23:52.684068 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.684073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.684085 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:23:52.684127 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.684129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.684146 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.684423 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:52.684508 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.685529 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.686743 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.687463 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.687785 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.687859 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.687890 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.687939 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260925 07:23:52.688442 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.683886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.4.47.3:43483)
0925 07:23:52.684076 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.684082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.684093 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:23:52.684132 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.684134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.684151 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.684236 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:52.684388 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.685125 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.686421 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.687327 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.687438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.687505 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.688296 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.688342 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260925 07:23:52.688760 20446 tablet_service.cc:1467] Processing CreateTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 (DEFAULT_TABLE table=test-table [id=cdd2bc37739844eb94f0d6c94f939fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:52.689088 20446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213f34ffa96d44f5afd86976e7cccbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:52.689106 20578 tablet_service.cc:1467] Processing CreateTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 (DEFAULT_TABLE table=test-table [id=cdd2bc37739844eb94f0d6c94f939fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:23:52.689437 20578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213f34ffa96d44f5afd86976e7cccbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:23:52.691919 20654 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Bootstrap starting.
I20260925 07:23:52.692178 20655 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Bootstrap starting.
I20260925 07:23:52.692761 20654 tablet_bootstrap.cc:654] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:52.692816 20655 tablet_bootstrap.cc:654] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Neither blocks nor log segments found. Creating new log.
I20260925 07:23:52.693434 20654 log.cc:824] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:52.693434 20655 log.cc:824] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Log is configured to *not* fsync() on all Append() calls
I20260925 07:23:52.694213 20655 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: No bootstrap required, opened a new log
I20260925 07:23:52.694280 20655 ts_tablet_manager.cc:1397] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:23:52.694468 20654 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: No bootstrap required, opened a new log
I20260925 07:23:52.694527 20654 ts_tablet_manager.cc:1397] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:23:52.696456 20654 raft_consensus.cc:348] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.696416 20655 raft_consensus.cc:348] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.696606 20654 raft_consensus.cc:374] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:52.696606 20655 raft_consensus.cc:374] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:23:52.696642 20654 raft_consensus.cc:729] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d802ab230840f3b62b2af571c25fb4, State: Initialized, Role: FOLLOWER
I20260925 07:23:52.696642 20655 raft_consensus.cc:729] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f9b20eb5b54c56a38da2e547f091ab, State: Initialized, Role: FOLLOWER
I20260925 07:23:52.696781 20655 consensus_queue.cc:260] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.696781 20654 consensus_queue.cc:260] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.697073 20654 ts_tablet_manager.cc:1428] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:23:52.697073 20655 ts_tablet_manager.cc:1428] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:23:52.697196 20644 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
I20260925 07:23:52.697222 20512 heartbeater.cc:503] Master 127.4.47.62:33747 was elected leader, sending a full tablet report...
W20260925 07:23:52.760079 20513 tablet.cc:2369] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:23:52.882341 20375 tablet.cc:2369] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:23:52.899400 20645 tablet.cc:2369] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:23:52.910599 20659 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:23:52.910732 20659 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.911044 20659 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637)
I20260925 07:23:52.914309 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.911305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43483 (local address 127.4.47.2:41111)
0925 07:23:52.911367 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.911382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.911453 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.911674 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.911675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.911677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:52.911754 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.911756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.912493 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.912495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.913372 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.913375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.914035 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.914036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.914120 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.914124 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.914186 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.914226 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:23:52.914299 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.911380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:41111 (local address 127.4.47.3:43483)
0925 07:23:52.911459 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.911462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.911491 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:23:52.911521 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.911522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.911533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.911622 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:52.911854 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.912385 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.913466 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.913965 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.914018 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.914027 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.914198 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.914228 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 07:23:52.914763 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.911427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.4.47.2:51021)
0925 07:23:52.911594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.911603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:23:52.911643 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:23:52.911854 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:23:52.911854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:23:52.911856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:23:52.911940 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.911941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.912771 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.912772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.913674 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:23:52.913677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.914516 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.914517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:23:52.914587 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.914591 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:23:52.914664 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 07:23:52.914702 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260925 07:23:52.914695 20598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213f34ffa96d44f5afd86976e7cccbd8" candidate_uuid: "81d802ab230840f3b62b2af571c25fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f9b20eb5b54c56a38da2e547f091ab" is_pre_election: true
I20260925 07:23:52.914801 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.911506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:51021 (local address 127.4.47.1:40637)
0925 07:23:52.911608 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.911624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.911637 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:23:52.911713 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.911714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.911747 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.911843 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:23:52.912035 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.912666 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.913768 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.914291 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.914358 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.914452 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.914671 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.914732 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
I20260925 07:23:52.914892 20598 raft_consensus.cc:2454] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d802ab230840f3b62b2af571c25fb4 in term 0.
I20260925 07:23:52.915047 20328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213f34ffa96d44f5afd86976e7cccbd8" candidate_uuid: "81d802ab230840f3b62b2af571c25fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" is_pre_election: true
I20260925 07:23:52.915133 20399 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4; no voters:
I20260925 07:23:52.915225 20328 raft_consensus.cc:2454] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d802ab230840f3b62b2af571c25fb4 in term 0.
I20260925 07:23:52.915290 20659 raft_consensus.cc:2781] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:23:52.915365 20659 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:23:52.915396 20659 raft_consensus.cc:3037] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:52.916129 20659 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.916289 20659 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637)
I20260925 07:23:52.916488 20328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213f34ffa96d44f5afd86976e7cccbd8" candidate_uuid: "81d802ab230840f3b62b2af571c25fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
I20260925 07:23:52.916476 20598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213f34ffa96d44f5afd86976e7cccbd8" candidate_uuid: "81d802ab230840f3b62b2af571c25fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f9b20eb5b54c56a38da2e547f091ab"
I20260925 07:23:52.916555 20328 raft_consensus.cc:3037] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:52.916574 20598 raft_consensus.cc:3037] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:23:52.917407 20328 raft_consensus.cc:2454] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d802ab230840f3b62b2af571c25fb4 in term 1.
I20260925 07:23:52.917533 20598 raft_consensus.cc:2454] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d802ab230840f3b62b2af571c25fb4 in term 1.
I20260925 07:23:52.917580 20399 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81d802ab230840f3b62b2af571c25fb4, b8c2d35f784f4d0d981ded1fb9d44257; no voters:
I20260925 07:23:52.917687 20659 raft_consensus.cc:2781] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:23:52.917860 20659 raft_consensus.cc:686] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 81d802ab230840f3b62b2af571c25fb4, State: Running, Role: LEADER
I20260925 07:23:52.917961 20659 consensus_queue.cc:237] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:23:52.918784 20183 catalog_manager.cc:5495] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "81d802ab230840f3b62b2af571c25fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } health_report { overall_health: UNKNOWN } } }
I20260925 07:23:52.977839 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.975290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.4.47.2:41411)
0925 07:23:52.975356 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:23:52.975358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:23:52.975367 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:23:52.975434 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:23:52.975435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:23:52.975456 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:23:52.975520 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:23:52.975759 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.976359 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.977090 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:23:52.977324 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:23:52.977381 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:23:52.977496 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:23:52.977594 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:23:52.977653 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:23:52.977683 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:23:52.977701 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:23:52.979786 20328 raft_consensus.cc:1261] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Refusing update from remote peer 81d802ab230840f3b62b2af571c25fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:52.979801 20598 raft_consensus.cc:1261] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Refusing update from remote peer 81d802ab230840f3b62b2af571c25fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:23:52.980159 20662 consensus_queue.cc:1035] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:52.980340 20659 consensus_queue.cc:1035] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:23:52.982201 20666 mvcc.cc:205] Tried to move back new op lower bound from 7333154951082078208 to 7333154950832656384. Current Snapshot: MvccSnapshot[applied={T|T < 7333154951082078208}]
I20260925 07:23:52.982507 20665 mvcc.cc:205] Tried to move back new op lower bound from 7333154951082078208 to 7333154950832656384. Current Snapshot: MvccSnapshot[applied={T|T < 7333154951082078208 or (T in {7333154951082078208})}]
I20260925 07:23:52.984371 20182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:52.984472 20182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:23:52.985164 20182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Sending DeleteTablet for 3 replicas of tablet 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:23:52.985491 20446 tablet_service.cc:1514] Processing DeleteTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:52 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:35230
I20260925 07:23:52.985507 20578 tablet_service.cc:1514] Processing DeleteTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:52 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:43834
I20260925 07:23:52.985685 20308 tablet_service.cc:1514] Processing DeleteTablet for tablet 213f34ffa96d44f5afd86976e7cccbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:23:52 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:53024
I20260925 07:23:52.985884 20676 tablet_replica.cc:330] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: stopping tablet replica
I20260925 07:23:52.985898 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20224
I20260925 07:23:52.985954 20676 raft_consensus.cc:2229] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:52.986023 20676 raft_consensus.cc:2258] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:52.991204 20674 tablet_replica.cc:330] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: stopping tablet replica
I20260925 07:23:52.991339 20674 raft_consensus.cc:2229] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:23:52.991464 20674 raft_consensus.cc:2258] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:23:52.991640 20675 tablet_replica.cc:330] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: stopping tablet replica
I20260925 07:23:52.991740 20675 raft_consensus.cc:2229] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:23:52.991838 20675 raft_consensus.cc:2258] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 07:23:52.991958 20170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:40637 (error 108)
I20260925 07:23:52.992103 20674 ts_tablet_manager.cc:1905] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:23:52.992276 20675 ts_tablet_manager.cc:1905] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:23:52.992579 20650 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:23:52.992426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40637 (local address 127.0.0.1:53040)
0925 07:23:52.992486 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.992546 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40637: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 07:23:52.992683 20170 catalog_manager.cc:4584] TS b8c2d35f784f4d0d981ded1fb9d44257 (127.4.47.1:40637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 213f34ffa96d44f5afd86976e7cccbd8: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40637: connect: Connection reset by peer (error 104)
I20260925 07:23:52.993403 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20379
I20260925 07:23:52.994321 20674 ts_tablet_manager.cc:1918] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:23:52.994371 20674 log.cc:1196] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:23:52.994652 20674 ts_tablet_manager.cc:1939] T 213f34ffa96d44f5afd86976e7cccbd8 P 23f9b20eb5b54c56a38da2e547f091ab: Deleting consensus metadata
I20260925 07:23:52.995266 20170 catalog_manager.cc:4841] TS 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): tablet 213f34ffa96d44f5afd86976e7cccbd8 (table test-table [id=cdd2bc37739844eb94f0d6c94f939fac]) successfully deleted
I20260925 07:23:52.998726 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20515
I20260925 07:23:52.998756 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:23:52.998641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41411 (local address 127.0.0.1:35246)
0925 07:23:52.998693 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:23:52.998721 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:41411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:23:53.004074 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20153
I20260925 07:24:13.009940 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--webserver_interface=127.4.47.62
--webserver_port=39885
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:13.093497 20678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:13.093801 20678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:13.093843 20678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:13.095381 20678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:24:13.095461 20678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:24:13.095482 20678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:24:13.095501 20678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:13.095606 20678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:13.095633 20678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:13.095661 20678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:13.095688 20678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:13.097421 20678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=39885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:13.097736 20678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:13.097981 20678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:13.103614 20684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.103714 20683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.103734 20686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:13.104080 20678 server_base.cc:1034] running on GCE node
I20260925 07:24:13.104393 20678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:13.104720 20678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:13.105894 20678 hybrid_clock.cc:648] HybridClock initialized: now 1790321053105881 us; error 54 us; skew 500 ppm
Sep 25 07:24:13 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:13.108551 20678 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:13.108981 20678 webserver.cc:460] Webserver started at http://127.4.47.62:39885/ using document root <none> and password file <none>
I20260925 07:24:13.109220 20678 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:13.109264 20678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:13.110661 20678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:13.111349 20693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.111586 20678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:13.111678 20678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "e0628f20f5374a798d577a13495caa7d"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "6b17412672b7e4c7e378d2fe8e7920f1"
server_key_iv: "027f0ad2047ac01098bd487eb34586fe"
server_key_version: "encryptionkey@0"
I20260925 07:24:13.112032 20678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:13.123457 20678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:13.125982 20678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:13.126125 20678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:13.130388 20678 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:33747
I20260925 07:24:13.130466 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:33747 every 8 connection(s)
I20260925 07:24:13.130872 20678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:13.132463 20746 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:24:13.133145 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap starting.
I20260925 07:24:13.135555 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20678
I20260925 07:24:13.135869 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:40637
--local_ip_for_outbound_sockets=127.4.47.1
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=32885
--webserver_interface=127.4.47.1
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:24:13.142436 20746 log.cc:824] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:13.144935 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:24:13.145306 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Bootstrap complete.
I20260925 07:24:13.147761 20746 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:24:13.148033 20746 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Initialized, Role: FOLLOWER
I20260925 07:24:13.148227 20746 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:24:13.148309 20746 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:13.148365 20746 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:13.148414 20746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:24:13.149471 20746 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:24:13.149601 20746 leader_election.cc:302] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0628f20f5374a798d577a13495caa7d; no voters:
I20260925 07:24:13.149809 20746 leader_election.cc:288] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 07:24:13.149845 20750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 07:24:13.149983 20750 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [term 3 LEADER]: Becoming Leader. State: Replica: e0628f20f5374a798d577a13495caa7d, State: Running, Role: LEADER
I20260925 07:24:13.150072 20750 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } }
I20260925 07:24:13.150101 20746 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:13.150362 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:24:13.150458 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:13.150656 20754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:13.151772 20751 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0628f20f5374a798d577a13495caa7d. Latest consensus state: current_term: 3 leader_uuid: "e0628f20f5374a798d577a13495caa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0628f20f5374a798d577a13495caa7d" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 33747 } } }
I20260925 07:24:13.151861 20751 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:13.152344 20754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:13.152508 20754 catalog_manager.cc:1259] Loaded cluster ID: f9dc68eaf56e4485a474ada50305c4cb
I20260925 07:24:13.152724 20754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:13.153538 20754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:13.153738 20754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Loaded TSK: 0
I20260925 07:24:13.153800 20754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0628f20f5374a798d577a13495caa7d: Loaded TSK: 1
F20260925 07:24:13.154034 20754 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f42aeb229ad google::LogMessage::Fail() at ??:0
@ 0x7f42aeb2588a google::LogMessage::SendToLog() at ??:0
@ 0x7f42aeb224de google::LogMessage::Flush() at ??:0
@ 0x7f42aeb23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f42b128f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f42b128cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f42b6a811cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f42b6a8148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f42b6a81594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f42b6a47c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f42b6a51ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f42b09f00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f42b09e5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f42b18496db start_thread at ??:0
@ 0x7f42ad62061f clone at ??:0
W20260925 07:24:13.243542 20748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:13.243781 20748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:13.243810 20748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:13.245350 20748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:13.245421 20748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:13.245481 20748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:13.247489 20748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:40637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=32885
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:13.247781 20748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:13.248040 20748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:13.253527 20787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.253541 20790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.253535 20788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:13.253901 20748 server_base.cc:1034] running on GCE node
I20260925 07:24:13.254146 20748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:13.254436 20748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:13.255633 20748 hybrid_clock.cc:648] HybridClock initialized: now 1790321053255619 us; error 28 us; skew 500 ppm
Sep 25 07:24:13 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:13.258394 20748 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:24:13.258872 20748 webserver.cc:460] Webserver started at http://127.4.47.1:32885/ using document root <none> and password file <none>
I20260925 07:24:13.259181 20748 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:13.259255 20748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:13.260771 20748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:13.261600 20797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.261785 20748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:13.261886 20748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "b8c2d35f784f4d0d981ded1fb9d44257"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "7e334b12b3b468e35e50ccbf9a3d1e1d"
server_key_iv: "7127be4ff6593a9c427484e79015e2a0"
server_key_version: "encryptionkey@0"
I20260925 07:24:13.262283 20748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:13.271013 20748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:13.273298 20748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:13.273473 20748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:13.273711 20748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:13.274184 20804 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:24:13.276161 20748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:24:13.276238 20748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:24:13.276286 20748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:24:13.276957 20748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:24:13.277016 20748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:24:13.277087 20804 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap starting.
I20260925 07:24:13.284853 20748 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:40637
I20260925 07:24:13.285004 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:40637 every 8 connection(s)
I20260925 07:24:13.285311 20748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:24:13.289098 20804 log.cc:824] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:13.291828 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20748
I20260925 07:24:13.292022 20804 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:24:13.292183 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:41411
--local_ip_for_outbound_sockets=127.4.47.2
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=42071
--webserver_interface=127.4.47.2
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:24:13.292377 20804 tablet_bootstrap.cc:492] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Bootstrap complete.
I20260925 07:24:13.293604 20804 ts_tablet_manager.cc:1397] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260925 07:24:13.295262 20804 raft_consensus.cc:348] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:13.295658 20804 raft_consensus.cc:729] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d35f784f4d0d981ded1fb9d44257, State: Initialized, Role: FOLLOWER
I20260925 07:24:13.295851 20804 consensus_queue.cc:260] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:13.296165 20804 ts_tablet_manager.cc:1428] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 07:24:13.357488 20914 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:24:13.286105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:58061)
0925 07:24:13.286308 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.286347 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:13.286910 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:13.357274 (+ 70364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
W20260925 07:24:13.357760 20912 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33747 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33747: Client connection negotiation failed: client connection to 127.4.47.62:33747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:24:13.358112 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.357998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:48989)
0925 07:24:13.358049 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.358091 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 07:24:13.358515 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.358426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:57505)
0925 07:24:13.358473 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.358498 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 07:24:13.358688 20912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:24:13.379966 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:13.380189 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:13.380215 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:13.381418 20916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:13.381461 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:13.381488 20916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:13.382972 20916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:41411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=42071
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:13.383260 20916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:13.383512 20916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:13.388679 20922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.388689 20923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.388689 20925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:13.388927 20916 server_base.cc:1034] running on GCE node
I20260925 07:24:13.389151 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:13.389408 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:13.390609 20916 hybrid_clock.cc:648] HybridClock initialized: now 1790321053390526 us; error 39 us; skew 500 ppm
Sep 25 07:24:13 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:13.393143 20916 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:24:13.393631 20916 webserver.cc:460] Webserver started at http://127.4.47.2:42071/ using document root <none> and password file <none>
I20260925 07:24:13.393883 20916 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:13.393952 20916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:13.395607 20916 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:13.396365 20932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.396535 20916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:13.396687 20916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "81d802ab230840f3b62b2af571c25fb4"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "2e01298e09d6b84696607d786d3342b5"
server_key_iv: "a918d0319b5c650466d628c1b3e476de"
server_key_version: "encryptionkey@0"
I20260925 07:24:13.397048 20916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:13.417665 20916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:13.420508 20916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:13.420704 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:13.420979 20916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:13.421470 20939 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 07:24:13.422878 20939 ts_tablet_manager.cc:1852] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 07:24:13.423235 20939 ts_tablet_manager.cc:1905] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:13.425514 20939 ts_tablet_manager.cc:1918] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:13.425590 20939 log.cc:1196] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:13.425803 20939 ts_tablet_manager.cc:1939] T 213f34ffa96d44f5afd86976e7cccbd8 P 81d802ab230840f3b62b2af571c25fb4: Deleting consensus metadata
I20260925 07:24:13.425959 20916 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 07:24:13.426028 20916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 07:24:13.426076 20916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:13.426102 20916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.432476 20916 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:41411
I20260925 07:24:13.432559 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:41411 every 8 connection(s)
I20260925 07:24:13.432916 20916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:24:13.434096 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.433637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:32843)
0925 07:24:13.433846 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.433927 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
W20260925 07:24:13.434321 21047 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33747 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33747: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
I20260925 07:24:13.434612 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.434520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:38157)
0925 07:24:13.434568 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.434595 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:24:13.434849 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.434799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:59985)
0925 07:24:13.434820 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.434838 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 07:24:13.434940 21047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:24:13.438597 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 20916
I20260925 07:24:13.438894 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:43483
--local_ip_for_outbound_sockets=127.4.47.3
--tserver_master_addrs=127.4.47.62:33747
--webserver_port=39355
--webserver_interface=127.4.47.3
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:13.521713 21050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:13.521941 21050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:13.521967 21050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:13.523130 21050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:13.523170 21050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:13.523198 21050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:13.524739 21050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:43483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=39355
--tserver_master_addrs=127.4.47.62:33747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:13.524950 21050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:13.525144 21050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:13.530441 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.530448 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:13.530495 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:13.530719 21050 server_base.cc:1034] running on GCE node
I20260925 07:24:13.530964 21050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:13.531262 21050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:13.532428 21050 hybrid_clock.cc:648] HybridClock initialized: now 1790321053532381 us; error 63 us; skew 500 ppm
Sep 25 07:24:13 dist-test-slave-zw07 krb5kdc[19606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:13.535189 21050 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:24:13.535665 21050 webserver.cc:460] Webserver started at http://127.4.47.3:39355/ using document root <none> and password file <none>
I20260925 07:24:13.535916 21050 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:13.535987 21050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:13.537405 21050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:13.538054 21065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.538246 21050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:13.538358 21050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "23f9b20eb5b54c56a38da2e547f091ab"
format_stamp: "Formatted at 2026-09-25 07:23:31 on dist-test-slave-zw07"
server_key: "d88c5d4771c7758a20d76818a0fece14"
server_key_iv: "91011bb32b6d2b9d483243d90498fa33"
server_key_version: "encryptionkey@0"
I20260925 07:24:13.538725 21050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:13.552765 21050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:13.555426 21050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:13.555608 21050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:13.555871 21050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:13.556208 21050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:13.556264 21050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.556310 21050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:13.556334 21050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:13.562563 21050 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:43483
I20260925 07:24:13.562747 21178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:43483 every 8 connection(s)
I20260925 07:24:13.563031 21050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:13.564288 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21050
I20260925 07:24:13.564714 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.563842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:44685)
0925 07:24:13.564022 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.564161 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
W20260925 07:24:13.564950 21179 heartbeater.cc:644] Failed to heartbeat to 127.4.47.62:33747 (0 consecutive failures): Network error: Failed to ping master at 127.4.47.62:33747: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
I20260925 07:24:13.565274 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.565153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:47035)
0925 07:24:13.565210 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.565249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:24:13.565601 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:13.565541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:45527)
0925 07:24:13.565564 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:13.565586 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 07:24:13.565724 21179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:24:14.359537 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.359222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:38839)
0925 07:24:14.359364 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:14.359411 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260925 07:24:14.435532 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.435339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:51087)
0925 07:24:14.435452 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:14.435494 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260925 07:24:14.566325 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.566066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:34433)
0925 07:24:14.566242 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:14.566271 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
I20260925 07:24:14.643772 21186 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:14.643935 21186 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:14.644359 21186 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
Sep 25 07:24:14 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.2@KRBTEST.COM
Sep 25 07:24:14 dist-test-slave-zw07 krb5kdc[19606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321053, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.3@KRBTEST.COM
I20260925 07:24:14.657567 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.644538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:41411 (local address 127.4.47.1:52011)
0925 07:24:14.644740 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:14.644752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:14.644813 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:14.645946 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:14.645949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:14.646374 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:14.646655 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:14.646664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.648443 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.648444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:14.649730 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:14.649733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.650683 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.650684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:14.650831 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:14.651456 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:14.651468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:14.653359 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:14.655751 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:14.655758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:14.655760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:14.656009 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:14.656280 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:14.656281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:14.656415 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:14.656979 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:14.656992 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:14.657111 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 07:24:14.657231 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 07:24:14.657442 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":26}
I20260925 07:24:14.657591 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.644662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:52011 (local address 127.4.47.2:41411)
0925 07:24:14.644832 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:14.644845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:14.644869 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:24:14.645659 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:14.645662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:14.645739 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:14.645904 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:14.646817 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.648314 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.649833 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.650543 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.650700 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:14.653473 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:14.653477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:14.653477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:14.653555 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:14.655660 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656099 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:14.656103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:14.656103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:14.656212 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656574 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:14.656575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:14.656575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:14.656899 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:14.656984 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:14.657324 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 07:24:14.657468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260925 07:24:14.657606 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.644716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:59047 (local address 127.4.47.3:43483)
0925 07:24:14.644843 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:14.644854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:14.644948 (+ 94us) server_negotiation.cc:407] Connection header received
0925 07:24:14.645665 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:14.645669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:14.645740 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:14.645918 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:14.647193 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.648477 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.649940 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.650604 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.650703 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:14.654215 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:14.654220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:14.654220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:14.654233 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:14.656108 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656498 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:14.656501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:14.656501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:14.656617 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656868 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:14.656869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:14.656870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:14.657141 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:14.657200 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:14.657387 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 07:24:14.657506 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260925 07:24:14.657618 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:14.644548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43483 (local address 127.4.47.1:59047)
0925 07:24:14.644890 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:14.644898 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:14.644941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:14.646770 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:14.646770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:14.646959 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:14.647032 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:14.647034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.648567 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.648568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:14.649730 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:14.649733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:14.650799 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:14.650800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:14.650897 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:14.651561 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:14.651566 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:14.654091 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:14.656190 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:14.656191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:14.656408 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:14.656734 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:14.656735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:14.656736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:14.656797 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:14.657213 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:14.657213 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:14.657255 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 07:24:14.657322 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:24:14.657498 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":157,"thread_start_us":82,"threads_started":1}
W20260925 07:24:14.658686 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:14.659022 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:14.659116 20801 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:14.659238 21186 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:15.360683 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:15.360341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:54849)
0925 07:24:15.360549 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:15.360636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260925 07:24:15.436178 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:15.435963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:41067)
0925 07:24:15.436095 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:15.436134 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 07:24:15.567022 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:15.566795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:52337)
0925 07:24:15.566922 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:15.566979 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260925 07:24:16.265765 21193 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:16.265877 21193 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:16.266076 21193 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:24:16.266444 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:16.266539 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:16.266664 20801 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:16.266764 21193 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:16.361497 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:16.361237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:49899)
0925 07:24:16.361378 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:16.361422 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260925 07:24:16.436976 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:16.436746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:33545)
0925 07:24:16.436888 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:16.436931 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260925 07:24:16.567826 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:16.567597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:40265)
0925 07:24:16.567732 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:16.567781 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 07:24:17.362357 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:17.362099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:59627)
0925 07:24:17.362261 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:17.362308 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260925 07:24:17.437717 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:17.437447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:58617)
0925 07:24:17.437621 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:17.437666 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260925 07:24:17.568544 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:17.568349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:50435)
0925 07:24:17.568467 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:17.568501 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260925 07:24:18.363216 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:18.362915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:53665)
0925 07:24:18.363133 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:18.363170 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260925 07:24:18.384387 21201 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:18.384481 21201 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:18.384704 21201 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:24:18.385134 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:18.385223 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:18.385288 20801 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:18.385381 21201 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:18.438493 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:18.438249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:35287)
0925 07:24:18.438401 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:18.438445 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260925 07:24:18.569345 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:18.569098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:47979)
0925 07:24:18.569240 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:18.569295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260925 07:24:19.364075 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:19.363831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:45783)
0925 07:24:19.363980 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:19.364027 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260925 07:24:19.439381 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:19.439052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:49489)
0925 07:24:19.439275 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:19.439323 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":149,"threads_started":1}
I20260925 07:24:19.570196 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:19.569919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:47709)
0925 07:24:19.570091 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:19.570140 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260925 07:24:20.365011 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:20.364772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:34827)
0925 07:24:20.364914 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:20.364964 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260925 07:24:20.440150 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:20.439914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:36419)
0925 07:24:20.440053 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:20.440102 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260925 07:24:20.571221 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:20.570899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:53889)
0925 07:24:20.571055 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:20.571173 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260925 07:24:21.274505 21210 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:21.274621 21210 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:21.274806 21210 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:24:21.275269 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:21.275388 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:21.275461 20798 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:21.275554 21210 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:21.365867 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:21.365612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:45977)
0925 07:24:21.365751 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:21.365798 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260925 07:24:21.440964 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:21.440722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:58187)
0925 07:24:21.440861 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:21.440910 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260925 07:24:21.571985 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:21.571757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:33583)
0925 07:24:21.571882 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:21.571921 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 07:24:22.366583 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:22.366340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:41853)
0925 07:24:22.366466 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:22.366511 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 07:24:22.441722 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:22.441488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:59355)
0925 07:24:22.441638 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:22.441672 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 07:24:22.572690 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:22.572460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:45555)
0925 07:24:22.572599 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:22.572641 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260925 07:24:23.367424 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:23.367165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:49935)
0925 07:24:23.367320 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:23.367363 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260925 07:24:23.438866 21218 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:23.438999 21218 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:23.439245 21218 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:24:23.439718 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:23.439801 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:23.439836 20801 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:23.439918 21218 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:23.442435 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:23.442227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:45529)
0925 07:24:23.442350 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:23.442394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260925 07:24:23.573498 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:23.573229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:38089)
0925 07:24:23.573397 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:23.573448 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260925 07:24:24.368211 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:24.367977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:42911)
0925 07:24:24.368116 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:24.368164 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 07:24:24.443574 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:24.443344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:57293)
0925 07:24:24.443478 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:24.443525 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 07:24:24.574350 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:24.574144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:40713)
0925 07:24:24.574264 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:24.574302 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260925 07:24:24.996065 21224 raft_consensus.cc:482] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:24.996178 21224 raft_consensus.cc:504] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23f9b20eb5b54c56a38da2e547f091ab" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43483 } } peers { permanent_uuid: "81d802ab230840f3b62b2af571c25fb4" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 41411 } } peers { permanent_uuid: "b8c2d35f784f4d0d981ded1fb9d44257" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40637 } }
I20260925 07:24:24.996429 21224 leader_election.cc:288] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483), 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411)
W20260925 07:24:24.996773 20798 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23f9b20eb5b54c56a38da2e547f091ab (127.4.47.3:43483): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
W20260925 07:24:24.996912 20801 leader_election.cc:341] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81d802ab230840f3b62b2af571c25fb4 (127.4.47.2:41411): Not found: Tablet not found: 213f34ffa96d44f5afd86976e7cccbd8
I20260925 07:24:24.996968 20801 leader_election.cc:302] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2d35f784f4d0d981ded1fb9d44257; no voters: 23f9b20eb5b54c56a38da2e547f091ab, 81d802ab230840f3b62b2af571c25fb4
I20260925 07:24:24.997074 21224 raft_consensus.cc:2726] T 213f34ffa96d44f5afd86976e7cccbd8 P b8c2d35f784f4d0d981ded1fb9d44257 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:24:25.368959 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:25.368747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:37359)
0925 07:24:25.368873 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:25.368910 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260925 07:24:25.444269 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:25.444056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:60097)
0925 07:24:25.444176 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:25.444220 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 07:24:25.575183 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:25.574903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:40641)
0925 07:24:25.575050 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:25.575136 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260925 07:24:26.369704 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:26.369466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:60199)
0925 07:24:26.369612 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:26.369658 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260925 07:24:26.445097 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:26.444844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:45825)
0925 07:24:26.444994 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:26.445046 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260925 07:24:26.575946 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:26.575694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:52913)
0925 07:24:26.575850 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:26.575900 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260925 07:24:27.370599 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:27.370296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:39239)
0925 07:24:27.370497 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:27.370549 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
I20260925 07:24:27.445906 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:27.445645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:49339)
0925 07:24:27.445812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:27.445859 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260925 07:24:27.576673 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:27.576447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:54955)
0925 07:24:27.576588 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:27.576626 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260925 07:24:28.371451 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:28.371225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.1:32781)
0925 07:24:28.371378 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:28.371407 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260925 07:24:28.446766 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:28.446526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.2:33317)
0925 07:24:28.446661 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:28.446712 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260925 07:24:28.570886 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20748
I20260925 07:24:28.575752 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 20916
I20260925 07:24:28.577474 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:28.577266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:33747 (local address 127.4.47.3:44345)
0925 07:24:28.577394 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:28.577434 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.62:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260925 07:24:28.580314 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21050
2026-09-25T07:24:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58585 ms)
[----------] 30 tests from SecurityITest (231445 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:28 dist-test-slave-zw07 krb5kdc[21239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:28 dist-test-slave-zw07 krb5kdc[21239](info): set up 2 sockets
Sep 25 07:24:28 dist-test-slave-zw07 krb5kdc[21239](info): commencing operation
krb5kdc: starting...
W20260925 07:24:29.630738 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:29 dist-test-slave-zw07 krb5kdc[21239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:29.754637 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34595
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:29.839280 21252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:29.839543 21252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:29.839584 21252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:29.840852 21252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:29.840940 21252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:29.840966 21252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:29.840999 21252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:29.841023 21252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:29.842844 21252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:29.843195 21252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:29.843451 21252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:24:29.849012 21252 server_base.cc:1034] running on GCE node
W20260925 07:24:29.849025 21257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:29.848992 21258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:29.848992 21260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:29.849645 21252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:29.849924 21252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:29.851153 21252 hybrid_clock.cc:648] HybridClock initialized: now 1790321069851124 us; error 42 us; skew 500 ppm
Sep 25 07:24:29 dist-test-slave-zw07 krb5kdc[21239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321069, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:29.854372 21252 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:29.854878 21252 webserver.cc:460] Webserver started at http://127.4.47.62:33983/ using document root <none> and password file <none>
I20260925 07:24:29.855206 21252 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:29.855279 21252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:29.855451 21252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:29.856561 21252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "5fb350f04c57446897ad2455f69d6ab9"
format_stamp: "Formatted at 2026-09-25 07:24:29 on dist-test-slave-zw07"
server_key: "b6fabc2055283c58e027e8663fd9b769"
server_key_iv: "969123a982cd36ba5d717f773c63f28f"
server_key_version: "encryptionkey@0"
I20260925 07:24:29.856974 21252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "5fb350f04c57446897ad2455f69d6ab9"
format_stamp: "Formatted at 2026-09-25 07:24:29 on dist-test-slave-zw07"
server_key: "b6fabc2055283c58e027e8663fd9b769"
server_key_iv: "969123a982cd36ba5d717f773c63f28f"
server_key_version: "encryptionkey@0"
I20260925 07:24:29.858426 21252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:29.859295 21267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:29.859527 21252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:29.859637 21252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "5fb350f04c57446897ad2455f69d6ab9"
format_stamp: "Formatted at 2026-09-25 07:24:29 on dist-test-slave-zw07"
server_key: "b6fabc2055283c58e027e8663fd9b769"
server_key_iv: "969123a982cd36ba5d717f773c63f28f"
server_key_version: "encryptionkey@0"
I20260925 07:24:29.859750 21252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:29.867412 21252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:29.869377 21252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:29.869545 21252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:29.873835 21252 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:34595
I20260925 07:24:29.873924 21319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:34595 every 8 connection(s)
I20260925 07:24:29.874292 21252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:29.875279 21320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:29.877643 21320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: Bootstrap starting.
I20260925 07:24:29.878273 21320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:29.878573 21320 log.cc:824] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:29.879346 21320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: No bootstrap required, opened a new log
I20260925 07:24:29.879925 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21252
I20260925 07:24:29.880040 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:29.880255 4284 external_mini_cluster.cc:1428] Setting key 9cd0960a7f021672ca0dc24c15f39d43
I20260925 07:24:29.881608 21320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } }
I20260925 07:24:29.881750 21320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:29.881772 21320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb350f04c57446897ad2455f69d6ab9, State: Initialized, Role: FOLLOWER
I20260925 07:24:29.881917 21320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } }
I20260925 07:24:29.881992 21320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:29.882014 21320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:29.882082 21320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:29.882831 21320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } }
I20260925 07:24:29.882966 21320 leader_election.cc:302] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb350f04c57446897ad2455f69d6ab9; no voters:
I20260925 07:24:29.883216 21320 leader_election.cc:288] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:29.883249 21325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:29.883373 21325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb350f04c57446897ad2455f69d6ab9, State: Running, Role: LEADER
I20260925 07:24:29.883496 21320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:29.883528 21325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } }
I20260925 07:24:29.883996 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb350f04c57446897ad2455f69d6ab9. Latest consensus state: current_term: 1 leader_uuid: "5fb350f04c57446897ad2455f69d6ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } } }
I20260925 07:24:29.884095 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb350f04c57446897ad2455f69d6ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb350f04c57446897ad2455f69d6ab9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 34595 } } }
I20260925 07:24:29.884161 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:29.884235 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:29.884481 21334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:29.885070 21334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:29.887341 21334 catalog_manager.cc:1347] Generated new cluster ID: 855aef1ad00f4b44adc01f2c05fe4805
I20260925 07:24:29.887395 21334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:24:29 dist-test-slave-zw07 krb5kdc[21239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:29.892946 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:29.880803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.4.47.62:34595)
0925 07:24:29.880995 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:29.881006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:29.881039 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:24:29.881800 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:29.881811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:29.881904 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:29.882077 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:29.883032 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:29.884581 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:29.885798 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:29.886288 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:29.886406 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:29.889223 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:29.889234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:29.889234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:29.889246 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:29.891261 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:29.891649 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:29.891653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:29.891653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:29.891771 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:29.891988 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:29.891989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:29.891990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:29.892321 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:29.892499 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:29.892574 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:24:29.892728 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 07:24:29.909453 21334 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:29.910250 21334 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:29.915848 21334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: Generated new TSK 0
I20260925 07:24:29.916109 21334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:24:29.951344 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:30.035130 21346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.035365 21346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.035390 21346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.036614 21346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.036665 21346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.036734 21346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:30.038311 21346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.038580 21346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.038832 21346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.043823 21351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.043901 21354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.043829 21352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.044080 21346 server_base.cc:1034] running on GCE node
I20260925 07:24:30.044303 21346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.044569 21346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.045771 21346 hybrid_clock.cc:648] HybridClock initialized: now 1790321070045747 us; error 41 us; skew 500 ppm
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:30.048671 21346 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:24:30.049197 21346 webserver.cc:460] Webserver started at http://127.4.47.1:39571/ using document root <none> and password file <none>
I20260925 07:24:30.049441 21346 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.049490 21346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.049629 21346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.050743 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "7d11dfef17db40c9ad64898065d201cc"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "65b76b5f69ca763541751a7f4f72aa03"
server_key_iv: "6dfec4f9b39a193aec6aa4edac2532b6"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.051194 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "7d11dfef17db40c9ad64898065d201cc"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "65b76b5f69ca763541751a7f4f72aa03"
server_key_iv: "6dfec4f9b39a193aec6aa4edac2532b6"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.052631 21346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.053498 21361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.053700 21346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.053795 21346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "7d11dfef17db40c9ad64898065d201cc"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "65b76b5f69ca763541751a7f4f72aa03"
server_key_iv: "6dfec4f9b39a193aec6aa4edac2532b6"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.053848 21346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:30.067580 21346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:30.069620 21346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:30.069736 21346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:30.069934 21346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:30.070261 21346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:30.070295 21346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.070319 21346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:30.070365 21346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.076356 21346 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:40981
I20260925 07:24:30.076400 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:40981 every 8 connection(s)
I20260925 07:24:30.076751 21346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:30.086740 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.077640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:47519 (local address 127.4.47.62:34595)
0925 07:24:30.077684 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.077687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.077923 (+ 236us) server_negotiation.cc:407] Connection header received
0925 07:24:30.078469 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.078470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.078483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.078540 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.079606 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.080139 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.081191 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.081406 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.081492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.083795 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.083796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.083797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:30.083804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.085182 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.085781 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.085782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.085783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.085837 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.086063 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.086064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.086064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.086171 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.086225 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.086531 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.086613 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:24:30.086941 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.077599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34595 (local address 127.4.47.1:47519)
0925 07:24:30.077789 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.077827 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.078347 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.078614 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.078618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.079291 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:30.079494 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.079499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.080264 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.080265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.081082 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.081084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.081504 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.081505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.081631 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.082155 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:30.082164 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:30.083700 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:30.085444 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.085452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.085455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.085703 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.085896 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.085897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.085897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.086006 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.086229 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:30.086233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:30.086410 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.086505 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.086674 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260925 07:24:30.087013 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21346
I20260925 07:24:30.087201 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:30.087423 4284 external_mini_cluster.cc:1428] Setting key 4f9d417543e05c1f6b5f305565588029
I20260925 07:24:30.087764 21475 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34595
I20260925 07:24:30.087878 21475 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:30.088085 21475 heartbeater.cc:511] Master 127.4.47.62:34595 requested a full tablet report, sending...
I20260925 07:24:30.088642 21284 ts_manager.cc:194] Registered new tserver with Master: 7d11dfef17db40c9ad64898065d201cc (127.4.47.1:40981)
I20260925 07:24:30.089344 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:47519
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:24:30.112191 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:30.194983 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.195268 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.195308 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.196511 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.196585 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.196630 21480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:30.198146 21480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.198431 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.198670 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.203912 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.203888 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.204013 21480 server_base.cc:1034] running on GCE node
W20260925 07:24:30.203888 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.204363 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.204639 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.205804 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790321070205798 us; error 38 us; skew 500 ppm
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:30.208387 21480 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:24:30.208892 21480 webserver.cc:460] Webserver started at http://127.4.47.2:45965/ using document root <none> and password file <none>
I20260925 07:24:30.209127 21480 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.209193 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.209342 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.210433 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "e594fb4059e746c9b1dad7f897ed2958"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "06415f85bcb9968f32797453c3d89487"
server_key_iv: "0660fbe3e99f052971ebace60cf992bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.210810 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "e594fb4059e746c9b1dad7f897ed2958"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "06415f85bcb9968f32797453c3d89487"
server_key_iv: "0660fbe3e99f052971ebace60cf992bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.212229 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:24:30.213025 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.213199 21480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:30.213297 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "e594fb4059e746c9b1dad7f897ed2958"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "06415f85bcb9968f32797453c3d89487"
server_key_iv: "0660fbe3e99f052971ebace60cf992bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.213380 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:30.241580 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:30.243789 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:30.243970 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:30.244225 21480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:30.244550 21480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:30.244603 21480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.244648 21480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:30.244674 21480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.250680 21480 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:33459
I20260925 07:24:30.250725 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:33459 every 8 connection(s)
I20260925 07:24:30.251053 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:30.257480 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21480
I20260925 07:24:30.257603 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:30.257808 4284 external_mini_cluster.cc:1428] Setting key 2c6b75af9693bca518535e79e9f2bead
I20260925 07:24:30.261138 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.251901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:47457 (local address 127.4.47.62:34595)
0925 07:24:30.251947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.251950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.252171 (+ 221us) server_negotiation.cc:407] Connection header received
0925 07:24:30.252652 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.252654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.252665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.252726 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.253648 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.254159 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.255274 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.255533 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.255599 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.257915 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.257916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.257917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:30.257925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.259559 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.260109 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.260110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.260111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.260183 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.260465 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.260466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.260466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.260584 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.260658 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.260953 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.261038 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:24:30.261364 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.251885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34595 (local address 127.4.47.2:47457)
0925 07:24:30.252042 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.252077 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.252550 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.252780 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.252783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.253351 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:30.253551 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.253556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.254265 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.254266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.255166 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.255169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.255618 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.255618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.255719 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.256206 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:30.256214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:30.257771 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:30.259638 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.259646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.259649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.260002 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.260252 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.260253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.260254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.260380 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.260649 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:30.260656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:30.260812 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.260929 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.261108 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260925 07:24:30.261898 21609 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34595
I20260925 07:24:30.262017 21609 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:30.262210 21609 heartbeater.cc:511] Master 127.4.47.62:34595 requested a full tablet report, sending...
I20260925 07:24:30.262665 21284 ts_manager.cc:194] Registered new tserver with Master: e594fb4059e746c9b1dad7f897ed2958 (127.4.47.2:33459)
I20260925 07:24:30.263206 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:47457
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:24:30.286319 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:30.368094 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.368391 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.368466 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.369635 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.369704 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.369750 21614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:30.371249 21614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.371523 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.371752 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.376924 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.376924 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.376932 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.377039 21614 server_base.cc:1034] running on GCE node
I20260925 07:24:30.377422 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.377681 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.378829 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790321070378804 us; error 38 us; skew 500 ppm
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:30.381538 21614 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:24:30.382031 21614 webserver.cc:460] Webserver started at http://127.4.47.3:37299/ using document root <none> and password file <none>
I20260925 07:24:30.382269 21614 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.382346 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.382496 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.383683 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "204da60352b9a6ff365da90dffd6bc18"
server_key_iv: "300100370eac71ab17e180fd9eca9cb1"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.384109 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "204da60352b9a6ff365da90dffd6bc18"
server_key_iv: "300100370eac71ab17e180fd9eca9cb1"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.385707 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:30.386509 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.386689 21614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.386792 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "204da60352b9a6ff365da90dffd6bc18"
server_key_iv: "300100370eac71ab17e180fd9eca9cb1"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.386883 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:30.399506 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:30.401554 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:30.401716 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:30.401957 21614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:30.402279 21614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:30.402341 21614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.402383 21614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:30.402410 21614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.408658 21614 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:38119
I20260925 07:24:30.408702 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:38119 every 8 connection(s)
I20260925 07:24:30.409029 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:30.411482 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21614
I20260925 07:24:30.411586 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:30.411787 4284 external_mini_cluster.cc:1428] Setting key 0a678c2978938cd51c778327d5fc9632
Sep 25 07:24:30 dist-test-slave-zw07 krb5kdc[21239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:30.418700 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.409795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:46171 (local address 127.4.47.62:34595)
0925 07:24:30.409850 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.409851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.410037 (+ 186us) server_negotiation.cc:407] Connection header received
0925 07:24:30.410585 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.410587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.410598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.410664 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.411822 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.412659 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.413655 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.413845 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.413907 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.416195 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.416197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.416197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:30.416204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.417556 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.417917 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.417918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.417919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.417970 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.418147 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.418148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.418148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.418250 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.418299 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.418571 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.418633 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:24:30.418916 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.409745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:34595 (local address 127.4.47.3:46171)
0925 07:24:30.409905 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.409947 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.410483 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.410741 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.410745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.411444 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:30.411701 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.411707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.412744 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.412745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.413568 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.413570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.413924 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.413926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.414029 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.414516 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:30.414525 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:30.416101 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:30.417626 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.417632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.417634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.417848 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.418011 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:30.418012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:30.418012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:30.418098 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:30.418334 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:30.418338 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:30.418457 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.418544 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.418664 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260925 07:24:30.419466 21743 heartbeater.cc:347] Connected to a master server at 127.4.47.62:34595
I20260925 07:24:30.419554 21743 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:30.419696 21743 heartbeater.cc:511] Master 127.4.47.62:34595 requested a full tablet report, sending...
I20260925 07:24:30.420035 21284 ts_manager.cc:194] Registered new tserver with Master: 2eb4cd572d7d49de9c1d1cfd2ff89008 (127.4.47.3:38119)
I20260925 07:24:30.420447 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:46171
I20260925 07:24:30.423545 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:30.431145 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.424954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.4.47.62:34595)
0925 07:24:30.425030 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.425033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.425131 (+ 98us) server_negotiation.cc:407] Connection header received
0925 07:24:30.425210 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.425211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.425221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.425298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.425863 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.426344 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.427099 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.427288 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.427367 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.428590 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.428592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.428592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:30.428600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.429869 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.430234 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.430235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.430236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.430297 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:30.430506 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:30.430507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:30.430508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:30.430665 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.430773 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.430916 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.431007 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:24:30.432878 21284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:30.434484 21284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:30.441820 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.4.47.3:38119)
0925 07:24:30.437346 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.437351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.437482 (+ 131us) server_negotiation.cc:407] Connection header received
0925 07:24:30.437669 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.437671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.437689 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.437750 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:30.438065 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.439281 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.440683 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.441314 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.441432 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.441497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.441680 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.441722 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":144,"threads_started":1}
I20260925 07:24:30.441821 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:38119 (local address 127.0.0.1:34922)
0925 07:24:30.437371 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.437418 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.437508 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.437867 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.437868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.437869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:30.437945 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.437947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.439402 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.439403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.440578 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.440583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.441436 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.441438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.441535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.441549 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.441653 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.441723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":71,"thread_start_us":50,"threads_started":1}
I20260925 07:24:30.441839 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.4.47.2:33459)
0925 07:24:30.438097 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.438100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.438108 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:30.438137 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.438138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.438150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.438242 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:30.438345 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.439028 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.440642 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.441386 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.441448 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.441506 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.441708 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.441752 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":76,"threads_started":1}
I20260925 07:24:30.442103 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:33459 (local address 127.0.0.1:60636)
0925 07:24:30.437461 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.437471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.437528 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.438239 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.438240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.438242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:30.438288 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.438289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.439183 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.439184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.440530 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.440532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.441537 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.441538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.441610 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.441614 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.441662 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.442033 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":317,"thread_start_us":66,"threads_started":1}
I20260925 07:24:30.442687 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40981 (local address 127.0.0.1:46540)
0925 07:24:30.437394 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.437420 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.437489 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.437858 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.437862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.437865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:30.437945 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.437947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.440275 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.440276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.441346 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.441349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.442427 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.442428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.442509 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.442513 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.442565 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.442610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260925 07:24:30.442991 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.437164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.4.47.1:40981)
0925 07:24:30.437640 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.437645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.437653 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:30.437690 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.437692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.437706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.437776 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:30.438065 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.439289 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.441468 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.442331 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.442430 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.442488 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.442881 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.442931 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":105,"threads_started":1}
I20260925 07:24:30.443496 21677 tablet_service.cc:1467] Processing CreateTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 (DEFAULT_TABLE table=test-table [id=8647bc3f4f614074adbbb6c73ab95cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:30.443604 21409 tablet_service.cc:1467] Processing CreateTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 (DEFAULT_TABLE table=test-table [id=8647bc3f4f614074adbbb6c73ab95cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:30.443876 21677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0eca9feccfc4110bb1b2339af1fdbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:30.443988 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0eca9feccfc4110bb1b2339af1fdbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:30.446625 21761 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: Bootstrap starting.
I20260925 07:24:30.442992 21543 tablet_service.cc:1467] Processing CreateTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 (DEFAULT_TABLE table=test-table [id=8647bc3f4f614074adbbb6c73ab95cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:30.446615 21760 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Bootstrap starting.
I20260925 07:24:30.447113 21543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0eca9feccfc4110bb1b2339af1fdbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:30.447383 21761 tablet_bootstrap.cc:654] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:30.447715 21760 tablet_bootstrap.cc:654] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:30.447930 21761 log.cc:824] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:30.448382 21760 log.cc:824] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:30.448755 21761 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: No bootstrap required, opened a new log
I20260925 07:24:30.448835 21761 ts_tablet_manager.cc:1397] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:30.449301 21760 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: No bootstrap required, opened a new log
I20260925 07:24:30.449366 21760 ts_tablet_manager.cc:1397] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:24:30.450325 21761 raft_consensus.cc:348] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.450455 21761 raft_consensus.cc:374] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:30.450477 21761 raft_consensus.cc:729] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d11dfef17db40c9ad64898065d201cc, State: Initialized, Role: FOLLOWER
I20260925 07:24:30.450598 21761 consensus_queue.cc:260] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.450838 21761 ts_tablet_manager.cc:1428] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:30.450914 21475 heartbeater.cc:503] Master 127.4.47.62:34595 was elected leader, sending a full tablet report...
I20260925 07:24:30.451396 21764 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Bootstrap starting.
I20260925 07:24:30.451443 21760 raft_consensus.cc:348] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.451614 21760 raft_consensus.cc:374] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:30.451640 21760 raft_consensus.cc:729] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb4cd572d7d49de9c1d1cfd2ff89008, State: Initialized, Role: FOLLOWER
I20260925 07:24:30.451776 21760 consensus_queue.cc:260] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.452045 21764 tablet_bootstrap.cc:654] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:30.452076 21760 ts_tablet_manager.cc:1428] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 07:24:30.452116 21743 heartbeater.cc:503] Master 127.4.47.62:34595 was elected leader, sending a full tablet report...
I20260925 07:24:30.452602 21764 log.cc:824] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:30.453471 21764 tablet_bootstrap.cc:492] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: No bootstrap required, opened a new log
I20260925 07:24:30.453554 21764 ts_tablet_manager.cc:1397] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:30.454953 21764 raft_consensus.cc:348] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.455173 21764 raft_consensus.cc:374] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:30.455247 21764 raft_consensus.cc:729] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e594fb4059e746c9b1dad7f897ed2958, State: Initialized, Role: FOLLOWER
I20260925 07:24:30.455371 21764 consensus_queue.cc:260] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.455682 21764 ts_tablet_manager.cc:1428] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:30.455693 21609 heartbeater.cc:503] Master 127.4.47.62:34595 was elected leader, sending a full tablet report...
W20260925 07:24:30.501796 21610 tablet.cc:2369] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:30.513263 21765 raft_consensus.cc:482] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:30.513461 21765 raft_consensus.cc:504] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.513916 21765 leader_election.cc:288] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb4cd572d7d49de9c1d1cfd2ff89008 (127.4.47.3:38119), e594fb4059e746c9b1dad7f897ed2958 (127.4.47.2:33459)
I20260925 07:24:30.517175 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.514113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:38119 (local address 127.4.47.1:41381)
0925 07:24:30.514195 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.514207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.514253 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.514448 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.514449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.514451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:30.514535 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.514537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.515301 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.515302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.516115 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.516117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.516833 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.516834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.516927 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.516931 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.517031 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.517083 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 07:24:30.517251 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.514233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:41381 (local address 127.4.47.3:38119)
0925 07:24:30.514285 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.514287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.514298 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:24:30.514331 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.514332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.514345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.514443 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:30.514617 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.515197 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.516331 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.516739 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.516834 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.516845 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.517070 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.517141 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:24:30.517895 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.514159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:33459 (local address 127.4.47.1:55135)
0925 07:24:30.514420 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.514432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.514546 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.514708 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.514709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.514711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:30.514789 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.514793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.515584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.515585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.516570 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.516573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.517558 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.517559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.517650 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.517655 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.517732 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.517802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":17,"thread_start_us":129,"threads_started":1}
I20260925 07:24:30.517984 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0eca9feccfc4110bb1b2339af1fdbc4" candidate_uuid: "7d11dfef17db40c9ad64898065d201cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" is_pre_election: true
I20260925 07:24:30.518182 21697 raft_consensus.cc:2454] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d11dfef17db40c9ad64898065d201cc in term 0.
I20260925 07:24:30.518445 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.514268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:55135 (local address 127.4.47.2:33459)
0925 07:24:30.514313 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.514315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.514529 (+ 214us) server_negotiation.cc:407] Connection header received
0925 07:24:30.514622 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.514623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.514636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.514702 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:30.514875 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.515480 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.516680 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.517400 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.517488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.517501 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.518320 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.518373 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:24:30.518433 21363 leader_election.cc:302] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb4cd572d7d49de9c1d1cfd2ff89008, 7d11dfef17db40c9ad64898065d201cc; no voters:
I20260925 07:24:30.518587 21765 raft_consensus.cc:2781] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:30.518635 21765 raft_consensus.cc:482] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:30.518661 21765 raft_consensus.cc:3037] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:30.518746 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0eca9feccfc4110bb1b2339af1fdbc4" candidate_uuid: "7d11dfef17db40c9ad64898065d201cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e594fb4059e746c9b1dad7f897ed2958" is_pre_election: true
I20260925 07:24:30.518904 21563 raft_consensus.cc:2454] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d11dfef17db40c9ad64898065d201cc in term 0.
I20260925 07:24:30.519657 21765 raft_consensus.cc:504] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.519829 21765 leader_election.cc:288] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [CANDIDATE]: Term 1 election: Requested vote from peers 2eb4cd572d7d49de9c1d1cfd2ff89008 (127.4.47.3:38119), e594fb4059e746c9b1dad7f897ed2958 (127.4.47.2:33459)
I20260925 07:24:30.519994 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0eca9feccfc4110bb1b2339af1fdbc4" candidate_uuid: "7d11dfef17db40c9ad64898065d201cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e594fb4059e746c9b1dad7f897ed2958"
I20260925 07:24:30.520017 21697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0eca9feccfc4110bb1b2339af1fdbc4" candidate_uuid: "7d11dfef17db40c9ad64898065d201cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008"
I20260925 07:24:30.520077 21563 raft_consensus.cc:3037] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:30.520110 21697 raft_consensus.cc:3037] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:30.520886 21697 raft_consensus.cc:2454] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d11dfef17db40c9ad64898065d201cc in term 1.
I20260925 07:24:30.521014 21563 raft_consensus.cc:2454] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d11dfef17db40c9ad64898065d201cc in term 1.
I20260925 07:24:30.521121 21363 leader_election.cc:302] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb4cd572d7d49de9c1d1cfd2ff89008, 7d11dfef17db40c9ad64898065d201cc; no voters:
I20260925 07:24:30.521252 21765 raft_consensus.cc:2781] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:30.521416 21765 raft_consensus.cc:686] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 1 LEADER]: Becoming Leader. State: Replica: 7d11dfef17db40c9ad64898065d201cc, State: Running, Role: LEADER
I20260925 07:24:30.521533 21765 consensus_queue.cc:237] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } }
I20260925 07:24:30.522523 21283 catalog_manager.cc:5495] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d11dfef17db40c9ad64898065d201cc (127.4.47.1). New cstate: current_term: 1 leader_uuid: "7d11dfef17db40c9ad64898065d201cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d11dfef17db40c9ad64898065d201cc" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 40981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 } health_report { overall_health: UNKNOWN } } }
I20260925 07:24:30.534772 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.532266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.4.47.1:40981)
0925 07:24:30.532331 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.532333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.532341 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:30.532373 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.532374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.532393 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.532459 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:30.532639 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.533119 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.533958 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.534152 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.534221 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.534415 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:30.534526 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:30.534620 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.534657 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.534686 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:24:30.536785 21697 raft_consensus.cc:1261] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 1 FOLLOWER]: Refusing update from remote peer 7d11dfef17db40c9ad64898065d201cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:30.536860 21563 raft_consensus.cc:1261] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 1 FOLLOWER]: Refusing update from remote peer 7d11dfef17db40c9ad64898065d201cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:30.537218 21765 consensus_queue.cc:1035] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb4cd572d7d49de9c1d1cfd2ff89008" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:30.537398 21770 consensus_queue.cc:1035] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e594fb4059e746c9b1dad7f897ed2958" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:30.539659 21773 mvcc.cc:205] Tried to move back new op lower bound from 7333155104915755008 to 7333155104856973312. Current Snapshot: MvccSnapshot[applied={T|T < 7333155104915755008}]
I20260925 07:24:30.540952 21775 mvcc.cc:205] Tried to move back new op lower bound from 7333155104915755008 to 7333155104856973312. Current Snapshot: MvccSnapshot[applied={T|T < 7333155104915755008 or (T in {7333155104915755008})}]
I20260925 07:24:30.542421 21284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:30.542533 21284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:30.543437 21284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fb350f04c57446897ad2455f69d6ab9: Sending DeleteTablet for 3 replicas of tablet a0eca9feccfc4110bb1b2339af1fdbc4
I20260925 07:24:30.543821 21409 tablet_service.cc:1514] Processing DeleteTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:30 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:46540
I20260925 07:24:30.543851 21543 tablet_service.cc:1514] Processing DeleteTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:30 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:60636
I20260925 07:24:30.544126 21677 tablet_service.cc:1514] Processing DeleteTablet for tablet a0eca9feccfc4110bb1b2339af1fdbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:30 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:34922
I20260925 07:24:30.544236 21783 tablet_replica.cc:330] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc: stopping tablet replica
I20260925 07:24:30.544335 21783 raft_consensus.cc:2229] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:30.544443 21783 raft_consensus.cc:2258] T a0eca9feccfc4110bb1b2339af1fdbc4 P 7d11dfef17db40c9ad64898065d201cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:30.544507 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21346
I20260925 07:24:30.545179 21784 tablet_replica.cc:330] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: stopping tablet replica
I20260925 07:24:30.545307 21784 raft_consensus.cc:2229] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:30.545459 21784 raft_consensus.cc:2258] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:30.546087 21784 ts_tablet_manager.cc:1905] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:30.546588 21785 tablet_replica.cc:330] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: stopping tablet replica
I20260925 07:24:30.546738 21785 raft_consensus.cc:2229] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:30.546881 21785 raft_consensus.cc:2258] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:30.547899 21785 ts_tablet_manager.cc:1905] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:30.548442 21784 ts_tablet_manager.cc:1918] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:30.548521 21784 log.cc:1196] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/a0eca9feccfc4110bb1b2339af1fdbc4
I20260925 07:24:30.548811 21784 ts_tablet_manager.cc:1939] T a0eca9feccfc4110bb1b2339af1fdbc4 P e594fb4059e746c9b1dad7f897ed2958: Deleting consensus metadata
I20260925 07:24:30.549222 21268 catalog_manager.cc:4841] TS e594fb4059e746c9b1dad7f897ed2958 (127.4.47.2:33459): tablet a0eca9feccfc4110bb1b2339af1fdbc4 (table test-table [id=8647bc3f4f614074adbbb6c73ab95cc7]) successfully deleted
I20260925 07:24:30.549417 21785 ts_tablet_manager.cc:1918] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:30.549520 21785 log.cc:1196] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/a0eca9feccfc4110bb1b2339af1fdbc4
I20260925 07:24:30.549815 21785 ts_tablet_manager.cc:1939] T a0eca9feccfc4110bb1b2339af1fdbc4 P 2eb4cd572d7d49de9c1d1cfd2ff89008: Deleting consensus metadata
I20260925 07:24:30.550555 21269 catalog_manager.cc:4841] TS 2eb4cd572d7d49de9c1d1cfd2ff89008 (127.4.47.3:38119): tablet a0eca9feccfc4110bb1b2339af1fdbc4 (table test-table [id=8647bc3f4f614074adbbb6c73ab95cc7]) successfully deleted
W20260925 07:24:30.551491 21268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:40981 (error 108)
I20260925 07:24:30.551730 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21480
I20260925 07:24:30.552055 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.551924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:40981 (local address 127.0.0.1:46562)
0925 07:24:30.551974 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.552014 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 07:24:30.552170 21268 catalog_manager.cc:4584] TS 7d11dfef17db40c9ad64898065d201cc (127.4.47.1:40981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0eca9feccfc4110bb1b2339af1fdbc4: Network error: Client connection negotiation failed: client connection to 127.4.47.1:40981: connect: Connection refused (error 111)
I20260925 07:24:30.556993 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21614
I20260925 07:24:30.561583 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21252
2026-09-25T07:24:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T07:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:30Z Disabled control of system clock
I20260925 07:24:30.590006 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45339
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:30.672324 21790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.672559 21790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.672580 21790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.673712 21790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:30.673753 21790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.673766 21790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.673779 21790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:30.673790 21790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:30.675233 21790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:45339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.675455 21790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.675715 21790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.681136 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.681066 21798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.681066 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.681334 21790 server_base.cc:1034] running on GCE node
I20260925 07:24:30.681769 21790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.682108 21790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.683284 21790 hybrid_clock.cc:648] HybridClock initialized: now 1790321070683270 us; error 33 us; skew 500 ppm
I20260925 07:24:30.684715 21790 webserver.cc:460] Webserver started at http://127.4.47.62:38747/ using document root <none> and password file <none>
I20260925 07:24:30.685001 21790 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.685075 21790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.685230 21790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.686335 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "7d73a747fedd7ba2e70a8cd096ef41c3"
server_key_iv: "2af79d106e5b77cb0dac47768b69dbe5"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.686713 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "7d73a747fedd7ba2e70a8cd096ef41c3"
server_key_iv: "2af79d106e5b77cb0dac47768b69dbe5"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.688158 21790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:30.688958 21804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.689131 21790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:30.689236 21790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "7d73a747fedd7ba2e70a8cd096ef41c3"
server_key_iv: "2af79d106e5b77cb0dac47768b69dbe5"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.689314 21790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:30.708729 21790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:30.709093 21790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:30.709249 21790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:30.713441 21790 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:45339
I20260925 07:24:30.713474 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:45339 every 8 connection(s)
I20260925 07:24:30.713816 21790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:30.714587 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21790
I20260925 07:24:30.714672 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:30.714862 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:30.714898 4284 external_mini_cluster.cc:1428] Setting key 57598d6dd4f75188cd20a6fabcc56be9
I20260925 07:24:30.717949 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: Bootstrap starting.
I20260925 07:24:30.718652 21857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:30.718976 21857 log.cc:824] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:30.719849 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: No bootstrap required, opened a new log
I20260925 07:24:30.721024 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.715582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.4.47.62:45339)
0925 07:24:30.715774 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.715781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.715817 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:24:30.716354 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.716362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.716425 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.716536 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.716830 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.717735 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.718846 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.719472 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.719592 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.720334 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.720338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.720338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:30.720502 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.720534 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:30.720545 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.720597 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.720695 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.720781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260925 07:24:30.722208 21857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } }
I20260925 07:24:30.722368 21857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:30.722414 21857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ea3a47e5be4aff8fc0d0862e4fb7eb, State: Initialized, Role: FOLLOWER
I20260925 07:24:30.722537 21857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } }
I20260925 07:24:30.722635 21857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:30.722679 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:30.722738 21857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:30.723379 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } }
I20260925 07:24:30.723503 21857 leader_election.cc:302] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47ea3a47e5be4aff8fc0d0862e4fb7eb; no voters:
I20260925 07:24:30.723708 21857 leader_election.cc:288] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:30.723806 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:30.724004 21862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [term 1 LEADER]: Becoming Leader. State: Replica: 47ea3a47e5be4aff8fc0d0862e4fb7eb, State: Running, Role: LEADER
I20260925 07:24:30.724056 21857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:30.724135 21862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } }
I20260925 07:24:30.724565 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47ea3a47e5be4aff8fc0d0862e4fb7eb. Latest consensus state: current_term: 1 leader_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } } }
I20260925 07:24:30.724550 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ea3a47e5be4aff8fc0d0862e4fb7eb" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 45339 } } }
I20260925 07:24:30.724653 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:30.724658 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:30.724995 21868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:30.725557 21868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:30.727120 21868 catalog_manager.cc:1347] Generated new cluster ID: 0f7cc624aa714ff78080db4e7ea4002e
I20260925 07:24:30.727197 21868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:30.740513 21868 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:30.741063 21868 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:30.743852 21868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: Generated new TSK 0
I20260925 07:24:30.744060 21868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:30.747155 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--builtin_ntp_servers=127.4.47.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:30.831436 21881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.831696 21881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.831720 21881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.833016 21881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.833058 21881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.833086 21881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:30.834475 21881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.834721 21881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.835055 21881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.841019 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.841238 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.841401 21881 server_base.cc:1034] running on GCE node
W20260925 07:24:30.841537 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.841749 21881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.842020 21881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.843149 21881 hybrid_clock.cc:648] HybridClock initialized: now 1790321070843120 us; error 39 us; skew 500 ppm
I20260925 07:24:30.844583 21881 webserver.cc:460] Webserver started at http://127.4.47.1:36813/ using document root <none> and password file <none>
I20260925 07:24:30.844871 21881 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.844929 21881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.845058 21881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.846568 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "64e6b878bee34ed0846fda859e9d2cdf"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "b23f8b111ce35ad3d18e534403e9a29d"
server_key_iv: "18f3b2f759b3a8c1306a7d303d0857a4"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.846999 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "64e6b878bee34ed0846fda859e9d2cdf"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "b23f8b111ce35ad3d18e534403e9a29d"
server_key_iv: "18f3b2f759b3a8c1306a7d303d0857a4"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.848694 21881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.849649 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.849879 21881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.849988 21881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "64e6b878bee34ed0846fda859e9d2cdf"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "b23f8b111ce35ad3d18e534403e9a29d"
server_key_iv: "18f3b2f759b3a8c1306a7d303d0857a4"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.850082 21881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:30.879375 21881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:30.879703 21881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:30.879832 21881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:30.880044 21881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:30.880425 21881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:30.880460 21881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.880517 21881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:30.880543 21881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.886930 21881 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:42571
I20260925 07:24:30.887006 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:42571 every 8 connection(s)
I20260925 07:24:30.887378 21881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:24:30.892838 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.888188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:40821 (local address 127.4.47.62:45339)
0925 07:24:30.888255 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:30.888257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:30.888497 (+ 240us) server_negotiation.cc:407] Connection header received
0925 07:24:30.888998 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:30.888999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:30.889015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:30.889082 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:30.889838 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.890371 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.891421 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.891628 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.891690 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.892567 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:30.892569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:30.892569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:30.892574 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:30.892588 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:30.892592 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:30.892645 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:30.892740 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 07:24:30.892772 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:24:30.892993 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 21881
I20260925 07:24:30.893097 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:30.893179 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:30.888177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45339 (local address 127.4.47.1:40821)
0925 07:24:30.888366 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:30.888404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:30.888897 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:30.889166 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:30.889170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:30.889179 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:30.889727 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.889731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.890461 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.890461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.891328 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:30.891330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:30.891711 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:30.891712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:30.891825 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:30.892414 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:30.892415 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:30.892444 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:30.892469 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:30.892657 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:30.892662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:30.892664 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:24:30.892718 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 07:24:30.892858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260925 07:24:30.893365 4284 external_mini_cluster.cc:1428] Setting key 9815a13b36c970f9fba4796e29c388b7
I20260925 07:24:30.893682 22009 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45339
I20260925 07:24:30.893805 22009 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:30.894040 22009 heartbeater.cc:511] Master 127.4.47.62:45339 requested a full tablet report, sending...
I20260925 07:24:30.894466 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--builtin_ntp_servers=127.4.47.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:24:30.894636 21821 ts_manager.cc:194] Registered new tserver with Master: 64e6b878bee34ed0846fda859e9d2cdf (127.4.47.1:42571)
I20260925 07:24:30.895418 21821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.1:40821
W20260925 07:24:30.977797 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:30.978075 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:30.978114 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:30.979523 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:30.979594 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:30.979640 22012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:30.981779 22012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:30.982081 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:30.982414 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:30.987737 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.987706 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:30.987715 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:30.987761 22012 server_base.cc:1034] running on GCE node
I20260925 07:24:30.988150 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:30.988477 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:30.989624 22012 hybrid_clock.cc:648] HybridClock initialized: now 1790321070989611 us; error 28 us; skew 500 ppm
I20260925 07:24:30.990849 22012 webserver.cc:460] Webserver started at http://127.4.47.2:37945/ using document root <none> and password file <none>
I20260925 07:24:30.991168 22012 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:30.991295 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:30.991436 22012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:30.992611 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "cb2242017a5a447f91e010b5a0bf01a7"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "6298a59774396befada9d625e56c45d1"
server_key_iv: "fac022632b8d558ecdfaacfcda5dc4f9"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.993005 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "cb2242017a5a447f91e010b5a0bf01a7"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "6298a59774396befada9d625e56c45d1"
server_key_iv: "fac022632b8d558ecdfaacfcda5dc4f9"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.994326 22012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:30.995102 22026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:30.995357 22012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:30.995456 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "cb2242017a5a447f91e010b5a0bf01a7"
format_stamp: "Formatted at 2026-09-25 07:24:30 on dist-test-slave-zw07"
server_key: "6298a59774396befada9d625e56c45d1"
server_key_iv: "fac022632b8d558ecdfaacfcda5dc4f9"
server_key_version: "encryptionkey@0"
I20260925 07:24:30.995543 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:31.025878 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:31.026398 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:31.026672 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:31.026985 22012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:31.027441 22012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:31.027503 22012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:31.027549 22012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:31.027580 22012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:31.035732 22012 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:42147
I20260925 07:24:31.035872 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:42147 every 8 connection(s)
I20260925 07:24:31.036270 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:24:31.039805 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22012
I20260925 07:24:31.039947 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:31.040170 4284 external_mini_cluster.cc:1428] Setting key 48b28fbd5e1341c58783fc0fcf466ffb
I20260925 07:24:31.041239 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--builtin_ntp_servers=127.4.47.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:24:31.043620 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.037062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:35121 (local address 127.4.47.62:45339)
0925 07:24:31.037139 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.037141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.037408 (+ 267us) server_negotiation.cc:407] Connection header received
0925 07:24:31.038021 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.038022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.038038 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.038135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:31.038822 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.039650 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.041477 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.041766 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.041823 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.043263 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:31.043265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:31.043265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:31.043272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:31.043290 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:31.043295 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:31.043328 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.043478 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.043509 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:24:31.044210 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.037066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45339 (local address 127.4.47.2:35121)
0925 07:24:31.037260 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.037303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.037900 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.038191 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.038194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.038203 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:31.038720 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.038725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.039757 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.039758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.041366 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.041370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.041916 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.041917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.042073 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.043046 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:31.043047 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:31.043114 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:31.043149 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:31.043376 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:31.043383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:31.043387 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.043780 (+ 393us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.043933 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260925 07:24:31.044751 22140 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45339
I20260925 07:24:31.044855 22140 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:31.045083 22140 heartbeater.cc:511] Master 127.4.47.62:45339 requested a full tablet report, sending...
I20260925 07:24:31.045532 21821 ts_manager.cc:194] Registered new tserver with Master: cb2242017a5a447f91e010b5a0bf01a7 (127.4.47.2:42147)
I20260925 07:24:31.046105 21821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.2:35121
W20260925 07:24:31.132699 22143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:31.132998 22143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:31.133030 22143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:31.134505 22143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:31.134567 22143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:31.134596 22143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:31.136099 22143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:31.136375 22143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:31.136693 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:31.142179 22148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:31.142169 22149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:31.142177 22151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:31.142516 22143 server_base.cc:1034] running on GCE node
I20260925 07:24:31.142728 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:31.142935 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:31.144071 22143 hybrid_clock.cc:648] HybridClock initialized: now 1790321071144060 us; error 27 us; skew 500 ppm
I20260925 07:24:31.145335 22143 webserver.cc:460] Webserver started at http://127.4.47.3:36617/ using document root <none> and password file <none>
I20260925 07:24:31.145569 22143 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:31.145615 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:31.145754 22143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:31.146889 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "36e0a3b829ed4e889011a155a457caaf"
format_stamp: "Formatted at 2026-09-25 07:24:31 on dist-test-slave-zw07"
server_key: "5766d6ec3cedf33d0a73ba91102c18dd"
server_key_iv: "47d58fb8c36e8cb4f7b3abb331009305"
server_key_version: "encryptionkey@0"
I20260925 07:24:31.147296 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "36e0a3b829ed4e889011a155a457caaf"
format_stamp: "Formatted at 2026-09-25 07:24:31 on dist-test-slave-zw07"
server_key: "5766d6ec3cedf33d0a73ba91102c18dd"
server_key_iv: "47d58fb8c36e8cb4f7b3abb331009305"
server_key_version: "encryptionkey@0"
I20260925 07:24:31.148644 22143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:31.149449 22157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:31.149657 22143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:31.149758 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "36e0a3b829ed4e889011a155a457caaf"
format_stamp: "Formatted at 2026-09-25 07:24:31 on dist-test-slave-zw07"
server_key: "5766d6ec3cedf33d0a73ba91102c18dd"
server_key_iv: "47d58fb8c36e8cb4f7b3abb331009305"
server_key_version: "encryptionkey@0"
I20260925 07:24:31.149842 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:31.165863 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:31.166222 22143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:31.166380 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:31.166635 22143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:31.166987 22143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:31.167042 22143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:31.167096 22143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:31.167142 22143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:31.173817 22143 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:34633
I20260925 07:24:31.173899 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:34633 every 8 connection(s)
I20260925 07:24:31.174254 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:31.178125 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22143
I20260925 07:24:31.178237 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:31.178438 4284 external_mini_cluster.cc:1428] Setting key 7d4cfcc616c7d917205990bb3a0632f7
I20260925 07:24:31.179801 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.175040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:51025 (local address 127.4.47.62:45339)
0925 07:24:31.175135 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.175137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.175375 (+ 238us) server_negotiation.cc:407] Connection header received
0925 07:24:31.175860 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.175861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.175871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.175937 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:31.176579 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.177072 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.178134 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.178462 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.178542 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.179590 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:31.179591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:31.179591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:31.179595 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:31.179609 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:31.179612 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:31.179651 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.179719 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.179742 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 07:24:31.180055 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.175029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:45339 (local address 127.4.47.3:51025)
0925 07:24:31.175234 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.175281 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.175778 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.175993 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.175996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.176005 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:31.176506 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.176510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.177180 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.177181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.178045 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.178048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.178573 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.178574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.178743 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.179437 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:31.179438 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:31.179477 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:31.179511 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:31.179658 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:31.179661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:31.179663 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.179711 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.179792 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260925 07:24:31.180495 22271 heartbeater.cc:347] Connected to a master server at 127.4.47.62:45339
I20260925 07:24:31.180575 22271 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:31.180730 22271 heartbeater.cc:511] Master 127.4.47.62:45339 requested a full tablet report, sending...
I20260925 07:24:31.181053 21821 ts_manager.cc:194] Registered new tserver with Master: 36e0a3b829ed4e889011a155a457caaf (127.4.47.3:34633)
I20260925 07:24:31.181383 21821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.3:51025
I20260925 07:24:31.190281 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:31.195199 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.191995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.4.47.62:45339)
0925 07:24:31.192038 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.192040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.192198 (+ 158us) server_negotiation.cc:407] Connection header received
0925 07:24:31.192282 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.192283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.192294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.192345 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:31.192532 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.193101 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.193872 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.194092 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.194172 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.194901 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:31.194902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:31.194902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:31.194908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:31.194923 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:31.194927 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:31.194999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.195058 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.195106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:24:31.196947 21821 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:31.198568 21821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:31.205269 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.201381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.4.47.3:34633)
0925 07:24:31.201865 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.201870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.201879 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:24:31.201914 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.201916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.201931 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.202034 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:31.202154 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.203226 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.204314 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.204833 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.204927 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205024 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.205133 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.205176 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":150,"threads_started":1}
I20260925 07:24:31.205262 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.201170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34633 (local address 127.0.0.1:54120)
0925 07:24:31.201283 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.201291 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.201331 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.202033 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.202033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.202035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:31.202069 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.202070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.203309 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.203310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.204216 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.204219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.204926 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.204927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.205007 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205022 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.205117 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.205189 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260925 07:24:31.205456 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.200886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:42147 (local address 127.0.0.1:35338)
0925 07:24:31.201066 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.201106 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.201176 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.201876 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.201877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.201879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:31.201933 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.201935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.202839 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.202840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.204212 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.204215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.205192 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.205194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.205267 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205269 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.205301 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.205348 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260925 07:24:31.205456 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.200922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:42571 (local address 127.0.0.1:49156)
0925 07:24:31.201280 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.201291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.201350 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.201641 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.201644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.201646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:31.201727 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.201729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.203049 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.203050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.204212 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.204215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.205194 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.205194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.205263 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205268 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.205301 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.205348 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":9,"thread_start_us":86,"threads_started":1}
I20260925 07:24:31.205902 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.201043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.4.47.1:42571)
0925 07:24:31.201270 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.201274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.201402 (+ 128us) server_negotiation.cc:407] Connection header received
0925 07:24:31.201464 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.201467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.201484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.201556 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:31.201797 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.202963 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.204323 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.205082 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.205488 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205746 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.205777 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.205817 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260925 07:24:31.205916 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.201393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.4.47.2:42147)
0925 07:24:31.201686 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.201691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.201700 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:24:31.201751 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.201753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.201768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.201878 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:31.201990 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.202740 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.204324 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.205086 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.205677 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.205765 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.205788 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.205826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260925 07:24:31.206456 22205 tablet_service.cc:1467] Processing CreateTablet for tablet a6eabfda7b1d42dca97cacae83456c4a (DEFAULT_TABLE table=test-table [id=f384619eecf34d88abb259a8039f3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:31.206521 21943 tablet_service.cc:1467] Processing CreateTablet for tablet a6eabfda7b1d42dca97cacae83456c4a (DEFAULT_TABLE table=test-table [id=f384619eecf34d88abb259a8039f3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:31.206522 22074 tablet_service.cc:1467] Processing CreateTablet for tablet a6eabfda7b1d42dca97cacae83456c4a (DEFAULT_TABLE table=test-table [id=f384619eecf34d88abb259a8039f3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:31.206847 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eabfda7b1d42dca97cacae83456c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:31.206852 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eabfda7b1d42dca97cacae83456c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:31.206856 22074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6eabfda7b1d42dca97cacae83456c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:31.209450 22285 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Bootstrap starting.
I20260925 07:24:31.210191 22285 tablet_bootstrap.cc:654] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:31.210461 22287 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Bootstrap starting.
I20260925 07:24:31.210590 22285 log.cc:824] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:31.211148 22287 tablet_bootstrap.cc:654] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:31.211131 22286 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Bootstrap starting.
I20260925 07:24:31.211436 22285 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: No bootstrap required, opened a new log
I20260925 07:24:31.211452 22287 log.cc:824] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:31.211506 22285 ts_tablet_manager.cc:1397] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:24:31.211778 22286 tablet_bootstrap.cc:654] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:31.212095 22286 log.cc:824] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:31.212270 22287 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: No bootstrap required, opened a new log
I20260925 07:24:31.212482 22287 ts_tablet_manager.cc:1397] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:24:31.212870 22286 tablet_bootstrap.cc:492] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: No bootstrap required, opened a new log
I20260925 07:24:31.212932 22286 ts_tablet_manager.cc:1397] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:24:31.214226 22285 raft_consensus.cc:348] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.214408 22285 raft_consensus.cc:374] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:31.214443 22285 raft_consensus.cc:729] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e0a3b829ed4e889011a155a457caaf, State: Initialized, Role: FOLLOWER
I20260925 07:24:31.214596 22285 consensus_queue.cc:260] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.214856 22285 ts_tablet_manager.cc:1428] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 07:24:31.214982 22271 heartbeater.cc:503] Master 127.4.47.62:45339 was elected leader, sending a full tablet report...
I20260925 07:24:31.215009 22286 raft_consensus.cc:348] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.215214 22286 raft_consensus.cc:374] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:31.215260 22286 raft_consensus.cc:729] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2242017a5a447f91e010b5a0bf01a7, State: Initialized, Role: FOLLOWER
I20260925 07:24:31.215162 22287 raft_consensus.cc:348] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.215313 22287 raft_consensus.cc:374] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:31.215353 22287 raft_consensus.cc:729] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e6b878bee34ed0846fda859e9d2cdf, State: Initialized, Role: FOLLOWER
I20260925 07:24:31.215402 22286 consensus_queue.cc:260] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.215483 22287 consensus_queue.cc:260] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.215646 22286 ts_tablet_manager.cc:1428] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:31.215703 22140 heartbeater.cc:503] Master 127.4.47.62:45339 was elected leader, sending a full tablet report...
I20260925 07:24:31.215750 22287 ts_tablet_manager.cc:1428] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 07:24:31.215852 22009 heartbeater.cc:503] Master 127.4.47.62:45339 was elected leader, sending a full tablet report...
W20260925 07:24:31.287225 22141 tablet.cc:2369] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:31.388262 22010 tablet.cc:2369] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:31.424868 22272 tablet.cc:2369] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:31.462219 22292 raft_consensus.cc:482] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:31.462466 22292 raft_consensus.cc:504] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.462911 22292 leader_election.cc:288] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64e6b878bee34ed0846fda859e9d2cdf (127.4.47.1:42571), 36e0a3b829ed4e889011a155a457caaf (127.4.47.3:34633)
I20260925 07:24:31.466221 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.463161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:42571 (local address 127.4.47.2:54521)
0925 07:24:31.463415 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.463429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.463501 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.463658 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.463660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.463661 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:31.463746 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.463748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.464447 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.464448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.465363 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.465365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.465939 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.466018 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.466021 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.466089 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.466133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260925 07:24:31.466212 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.463241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:54521 (local address 127.4.47.1:42571)
0925 07:24:31.463419 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.463422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.463481 (+ 59us) server_negotiation.cc:407] Connection header received
0925 07:24:31.463570 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.463571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.463590 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.463655 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:31.463827 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.464343 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465468 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.465845 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465939 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.465950 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.466096 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.466133 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260925 07:24:31.466233 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.463464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:58301 (local address 127.4.47.3:34633)
0925 07:24:31.463517 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.463519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.463671 (+ 152us) server_negotiation.cc:407] Connection header received
0925 07:24:31.463750 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.463751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.463764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.463810 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:31.463967 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.464437 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465488 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.465857 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.465954 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.466124 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.466164 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:24:31.466240 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.463369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:34633 (local address 127.4.47.2:58301)
0925 07:24:31.463587 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.463602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:31.463688 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:31.463820 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:31.463821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:31.463822 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:31.463897 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.463898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.464528 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.464529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.465392 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:31.465395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.465948 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.465949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:31.466039 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.466042 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:31.466105 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 07:24:31.466145 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":51,"thread_start_us":108,"threads_started":1}
I20260925 07:24:31.466665 21963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eabfda7b1d42dca97cacae83456c4a" candidate_uuid: "cb2242017a5a447f91e010b5a0bf01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e6b878bee34ed0846fda859e9d2cdf" is_pre_election: true
I20260925 07:24:31.466701 22223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eabfda7b1d42dca97cacae83456c4a" candidate_uuid: "cb2242017a5a447f91e010b5a0bf01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e0a3b829ed4e889011a155a457caaf" is_pre_election: true
I20260925 07:24:31.466859 22223 raft_consensus.cc:2454] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb2242017a5a447f91e010b5a0bf01a7 in term 0.
I20260925 07:24:31.466859 21963 raft_consensus.cc:2454] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb2242017a5a447f91e010b5a0bf01a7 in term 0.
I20260925 07:24:31.467141 22028 leader_election.cc:302] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e0a3b829ed4e889011a155a457caaf, cb2242017a5a447f91e010b5a0bf01a7; no voters:
I20260925 07:24:31.467270 22292 raft_consensus.cc:2781] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:31.467362 22292 raft_consensus.cc:482] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:31.467407 22292 raft_consensus.cc:3037] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:31.468149 22292 raft_consensus.cc:504] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.468304 22292 leader_election.cc:288] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [CANDIDATE]: Term 1 election: Requested vote from peers 64e6b878bee34ed0846fda859e9d2cdf (127.4.47.1:42571), 36e0a3b829ed4e889011a155a457caaf (127.4.47.3:34633)
I20260925 07:24:31.468494 21963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eabfda7b1d42dca97cacae83456c4a" candidate_uuid: "cb2242017a5a447f91e010b5a0bf01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e6b878bee34ed0846fda859e9d2cdf"
I20260925 07:24:31.468508 22223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6eabfda7b1d42dca97cacae83456c4a" candidate_uuid: "cb2242017a5a447f91e010b5a0bf01a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36e0a3b829ed4e889011a155a457caaf"
I20260925 07:24:31.468583 22223 raft_consensus.cc:3037] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:31.468583 21963 raft_consensus.cc:3037] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:31.469601 22223 raft_consensus.cc:2454] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb2242017a5a447f91e010b5a0bf01a7 in term 1.
I20260925 07:24:31.469601 21963 raft_consensus.cc:2454] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb2242017a5a447f91e010b5a0bf01a7 in term 1.
I20260925 07:24:31.469786 22028 leader_election.cc:302] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64e6b878bee34ed0846fda859e9d2cdf, cb2242017a5a447f91e010b5a0bf01a7; no voters:
I20260925 07:24:31.469882 22292 raft_consensus.cc:2781] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:31.470075 22292 raft_consensus.cc:686] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 1 LEADER]: Becoming Leader. State: Replica: cb2242017a5a447f91e010b5a0bf01a7, State: Running, Role: LEADER
I20260925 07:24:31.470189 22292 consensus_queue.cc:237] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } }
I20260925 07:24:31.471283 21821 catalog_manager.cc:5495] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb2242017a5a447f91e010b5a0bf01a7 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "cb2242017a5a447f91e010b5a0bf01a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2242017a5a447f91e010b5a0bf01a7" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 } health_report { overall_health: UNKNOWN } } }
I20260925 07:24:31.491771 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.489155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.4.47.2:42147)
0925 07:24:31.489215 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:31.489218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:31.489226 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:31.489284 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:31.489284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:31.489303 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:31.489384 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:31.489538 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.490015 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.490850 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:31.491150 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:31.491228 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:31.491381 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:31.491497 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:31.491596 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:31.491615 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 07:24:31.491634 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:24:31.493794 22223 raft_consensus.cc:1261] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 1 FOLLOWER]: Refusing update from remote peer cb2242017a5a447f91e010b5a0bf01a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:31.493794 21963 raft_consensus.cc:1261] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 1 FOLLOWER]: Refusing update from remote peer cb2242017a5a447f91e010b5a0bf01a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:31.494179 22295 consensus_queue.cc:1035] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64e6b878bee34ed0846fda859e9d2cdf" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:31.494333 22292 consensus_queue.cc:1035] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e0a3b829ed4e889011a155a457caaf" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:31.497018 22300 mvcc.cc:205] Tried to move back new op lower bound from 7333155108835467264 to 7333155108742594560. Current Snapshot: MvccSnapshot[applied={T|T < 7333155108835467264}]
I20260925 07:24:31.497699 22299 mvcc.cc:205] Tried to move back new op lower bound from 7333155108835467264 to 7333155108742594560. Current Snapshot: MvccSnapshot[applied={T|T < 7333155108835467264}]
I20260925 07:24:31.499198 21820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:31.499315 21820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:31.500108 21820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47ea3a47e5be4aff8fc0d0862e4fb7eb: Sending DeleteTablet for 3 replicas of tablet a6eabfda7b1d42dca97cacae83456c4a
I20260925 07:24:31.500460 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eabfda7b1d42dca97cacae83456c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:31 UTC) from {username='slave'} at 127.0.0.1:49156
I20260925 07:24:31.500473 22074 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eabfda7b1d42dca97cacae83456c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:31 UTC) from {username='slave'} at 127.0.0.1:35338
I20260925 07:24:31.500720 22309 tablet_replica.cc:330] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf: stopping tablet replica
I20260925 07:24:31.500706 22205 tablet_service.cc:1514] Processing DeleteTablet for tablet a6eabfda7b1d42dca97cacae83456c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:31 UTC) from {username='slave'} at 127.0.0.1:54120
I20260925 07:24:31.500903 22309 raft_consensus.cc:2229] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:31.500926 22308 tablet_replica.cc:330] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: stopping tablet replica
I20260925 07:24:31.501004 22308 raft_consensus.cc:2229] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:31.501013 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21881
I20260925 07:24:31.501104 22308 raft_consensus.cc:2258] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:31.501186 22309 raft_consensus.cc:2258] T a6eabfda7b1d42dca97cacae83456c4a P 64e6b878bee34ed0846fda859e9d2cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:31.501569 22308 ts_tablet_manager.cc:1905] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:31.501586 22310 tablet_replica.cc:330] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: stopping tablet replica
I20260925 07:24:31.501744 22310 raft_consensus.cc:2229] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:31.501909 22310 raft_consensus.cc:2258] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:31.502418 22310 ts_tablet_manager.cc:1905] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:31.503600 22308 ts_tablet_manager.cc:1918] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:31.503679 22308 log.cc:1196] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/a6eabfda7b1d42dca97cacae83456c4a
I20260925 07:24:31.503955 22308 ts_tablet_manager.cc:1939] T a6eabfda7b1d42dca97cacae83456c4a P cb2242017a5a447f91e010b5a0bf01a7: Deleting consensus metadata
I20260925 07:24:31.504350 22310 ts_tablet_manager.cc:1918] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:31.504436 22310 log.cc:1196] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/a6eabfda7b1d42dca97cacae83456c4a
I20260925 07:24:31.504657 22310 ts_tablet_manager.cc:1939] T a6eabfda7b1d42dca97cacae83456c4a P 36e0a3b829ed4e889011a155a457caaf: Deleting consensus metadata
I20260925 07:24:31.504841 21807 catalog_manager.cc:4841] TS cb2242017a5a447f91e010b5a0bf01a7 (127.4.47.2:42147): tablet a6eabfda7b1d42dca97cacae83456c4a (table test-table [id=f384619eecf34d88abb259a8039f3a3c]) successfully deleted
I20260925 07:24:31.505059 21806 catalog_manager.cc:4841] TS 36e0a3b829ed4e889011a155a457caaf (127.4.47.3:34633): tablet a6eabfda7b1d42dca97cacae83456c4a (table test-table [id=f384619eecf34d88abb259a8039f3a3c]) successfully deleted
W20260925 07:24:31.509539 21806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:42571 (error 108)
I20260925 07:24:31.509828 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22012
I20260925 07:24:31.510077 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:31.509946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:42571 (local address 127.0.0.1:49162)
0925 07:24:31.510001 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:31.510038 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:24:31.511534 21806 catalog_manager.cc:4584] TS 64e6b878bee34ed0846fda859e9d2cdf (127.4.47.1:42571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6eabfda7b1d42dca97cacae83456c4a: Network error: Client connection negotiation failed: client connection to 127.4.47.1:42571: connect: Connection refused (error 111)
I20260925 07:24:31.514554 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22143
I20260925 07:24:31.519008 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 21790
2026-09-25T07:24:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (956 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:31 dist-test-slave-zw07 krb5kdc[22314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:31 dist-test-slave-zw07 krb5kdc[22314](info): set up 2 sockets
Sep 25 07:24:31 dist-test-slave-zw07 krb5kdc[22314](info): commencing operation
krb5kdc: starting...
W20260925 07:24:32.550897 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:32 dist-test-slave-zw07 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:32.674582 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40601
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:40601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:32.760881 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:32.761101 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:32.761124 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:32.762238 22327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:32.762274 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:32.762288 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:32.762300 22327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:32.762313 22327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:32.763827 22327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:40601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:32.764031 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:32.764202 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:32.769366 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:32.769387 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:32.769553 22327 server_base.cc:1034] running on GCE node
W20260925 07:24:32.769409 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:32.769893 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:32.770206 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:32.771360 22327 hybrid_clock.cc:648] HybridClock initialized: now 1790321072771332 us; error 42 us; skew 500 ppm
Sep 25 07:24:32 dist-test-slave-zw07 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321072, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:32.773955 22327 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:32.774483 22327 webserver.cc:460] Webserver started at http://127.4.47.62:34171/ using document root <none> and password file <none>
I20260925 07:24:32.774739 22327 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:32.774787 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:32.774960 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:32.776094 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "542c46be06414aaab689966d8b0dfa0b"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "e518f2a8df5b87561302c7fcfe4ef27a"
server_key_iv: "1c4e7ec90e7575adb8691bfad57dc852"
server_key_version: "encryptionkey@0"
I20260925 07:24:32.776479 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "542c46be06414aaab689966d8b0dfa0b"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "e518f2a8df5b87561302c7fcfe4ef27a"
server_key_iv: "1c4e7ec90e7575adb8691bfad57dc852"
server_key_version: "encryptionkey@0"
I20260925 07:24:32.777848 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:24:32.778601 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:32.778784 22327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:24:32.778882 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "542c46be06414aaab689966d8b0dfa0b"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "e518f2a8df5b87561302c7fcfe4ef27a"
server_key_iv: "1c4e7ec90e7575adb8691bfad57dc852"
server_key_version: "encryptionkey@0"
I20260925 07:24:32.778970 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:32.794476 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:32.796512 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:32.796701 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:32.801131 22327 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:40601
I20260925 07:24:32.801167 22394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:40601 every 8 connection(s)
I20260925 07:24:32.801541 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:32.802429 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:32.804853 22395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: Bootstrap starting.
I20260925 07:24:32.805485 22395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:32.805794 22395 log.cc:824] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:32.806581 22395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: No bootstrap required, opened a new log
I20260925 07:24:32.808318 22395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } }
I20260925 07:24:32.808471 22395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:32.808495 22395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542c46be06414aaab689966d8b0dfa0b, State: Initialized, Role: FOLLOWER
I20260925 07:24:32.808673 22395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } }
I20260925 07:24:32.808751 22395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:32.808784 22395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:32.808851 22395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:32.809429 22395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } }
I20260925 07:24:32.809556 22395 leader_election.cc:302] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 542c46be06414aaab689966d8b0dfa0b; no voters:
I20260925 07:24:32.809743 22395 leader_election.cc:288] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:32.809924 22399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:32.809998 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22327
I20260925 07:24:32.810009 22395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:32.810120 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:32.810322 22399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [term 1 LEADER]: Becoming Leader. State: Replica: 542c46be06414aaab689966d8b0dfa0b, State: Running, Role: LEADER
I20260925 07:24:32.810348 4284 external_mini_cluster.cc:1428] Setting key cf32d882f571ad7c3928edd6d464d850
I20260925 07:24:32.810478 22399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } }
I20260925 07:24:32.811261 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "542c46be06414aaab689966d8b0dfa0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } } }
I20260925 07:24:32.811383 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:32.811579 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 542c46be06414aaab689966d8b0dfa0b. Latest consensus state: current_term: 1 leader_uuid: "542c46be06414aaab689966d8b0dfa0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542c46be06414aaab689966d8b0dfa0b" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 40601 } } }
I20260925 07:24:32.811741 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:32.811805 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:32.812279 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:32.816139 22415 catalog_manager.cc:1347] Generated new cluster ID: a346432ff7484e709eb1373fba49bbc6
I20260925 07:24:32.816205 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:24:32 dist-test-slave-zw07 krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:32.823091 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:32.811097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.4.47.62:40601)
0925 07:24:32.811766 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:32.811777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:32.811810 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:24:32.811936 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:32.811946 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:32.811999 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:32.812170 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:32.812974 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:32.814423 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:32.815357 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:32.815867 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:32.816005 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:32.819139 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:32.819145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:32.819146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:32.819174 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:32.821291 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:32.821712 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:32.821715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:32.821716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:32.821849 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:32.822054 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:32.822055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:32.822056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:32.822408 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:32.822573 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:32.822681 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 07:24:32.822831 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":149,"threads_started":1}
I20260925 07:24:32.842262 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:32.842869 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:32.853422 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: Generated new TSK 0
I20260925 07:24:32.853677 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:24:32.886459 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:32.986424 22421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:32.986657 22421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:32.986685 22421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:32.988027 22421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:32.988088 22421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:32.988117 22421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:32.989910 22421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:32.990289 22421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:32.990579 22421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:32.996114 22427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:32.996109 22426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:32.996304 22421 server_base.cc:1034] running on GCE node
W20260925 07:24:32.996119 22429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:32.996593 22421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:32.996820 22421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:32.997972 22421 hybrid_clock.cc:648] HybridClock initialized: now 1790321072997957 us; error 28 us; skew 500 ppm
Sep 25 07:24:32 dist-test-slave-zw07 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321072, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:33.000720 22421 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:24:33.001199 22421 webserver.cc:460] Webserver started at http://127.4.47.1:33447/ using document root <none> and password file <none>
I20260925 07:24:33.001456 22421 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:33.001528 22421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:33.001677 22421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:33.002754 22421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "5261261250d1478d9af1903baeb29d94"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "889b77620e08213803363d9c8746489f"
server_key_iv: "13a6ae8fc4ddfe60788fc0c26e7d886b"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.003172 22421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "5261261250d1478d9af1903baeb29d94"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "889b77620e08213803363d9c8746489f"
server_key_iv: "13a6ae8fc4ddfe60788fc0c26e7d886b"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.004622 22421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:33.005484 22436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.005712 22421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:33.005818 22421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "5261261250d1478d9af1903baeb29d94"
format_stamp: "Formatted at 2026-09-25 07:24:32 on dist-test-slave-zw07"
server_key: "889b77620e08213803363d9c8746489f"
server_key_iv: "13a6ae8fc4ddfe60788fc0c26e7d886b"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.005904 22421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:33.043591 22421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:33.045663 22421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:33.045854 22421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:33.046111 22421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:33.046440 22421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:33.046495 22421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.046533 22421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:33.046581 22421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.052767 22421 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:34303
I20260925 07:24:33.052949 22549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:34303 every 8 connection(s)
I20260925 07:24:33.053243 22421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:24:33 dist-test-slave-zw07 krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321072, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:33.062899 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22421
I20260925 07:24:33.062968 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.054074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:55111 (local address 127.4.47.62:40601)
0925 07:24:33.054166 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.054169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.054336 (+ 167us) server_negotiation.cc:407] Connection header received
0925 07:24:33.054805 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.054806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.054820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.054874 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:33.055967 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.056487 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.057528 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.057761 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.057820 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.060305 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:33.060306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:33.060307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:33.060314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:33.061612 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.062019 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.062020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.062021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.062075 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.062298 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.062299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.062299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.062406 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:33.062459 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.062732 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.062843 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 07:24:33.063022 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:33.063043 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.054022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40601 (local address 127.4.47.1:55111)
0925 07:24:33.054203 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.054241 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.054716 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.054964 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.054968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.055637 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:33.055852 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.055857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.056575 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.056575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.057428 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.057430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.057844 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.057845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.057968 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.058599 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:33.058607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:33.060199 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:33.061695 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.061701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.061703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.061932 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.062142 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.062143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.062143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.062241 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.062463 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:33.062467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:33.062585 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.062686 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.062809 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260925 07:24:33.063271 4284 external_mini_cluster.cc:1428] Setting key a2b15d4824220b12291c17b6ad6c62b5
I20260925 07:24:33.063608 22550 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40601
I20260925 07:24:33.063711 22550 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:33.063910 22550 heartbeater.cc:511] Master 127.4.47.62:40601 requested a full tablet report, sending...
I20260925 07:24:33.064489 22359 ts_manager.cc:194] Registered new tserver with Master: 5261261250d1478d9af1903baeb29d94 (127.4.47.1:34303)
I20260925 07:24:33.065442 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:55111
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:24:33.091223 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:33.175045 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:33.175360 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:33.175403 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:33.176613 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:33.176692 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:33.176736 22555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:33.178215 22555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:33.178501 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:33.178738 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:33.185314 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.185314 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.185338 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:33.185550 22555 server_base.cc:1034] running on GCE node
I20260925 07:24:33.185822 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:33.186098 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:33.187238 22555 hybrid_clock.cc:648] HybridClock initialized: now 1790321073187221 us; error 39 us; skew 500 ppm
Sep 25 07:24:33 dist-test-slave-zw07 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:33.190018 22555 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:24:33.190478 22555 webserver.cc:460] Webserver started at http://127.4.47.2:39031/ using document root <none> and password file <none>
I20260925 07:24:33.190722 22555 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:33.190768 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:33.190929 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:33.192117 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "35ca8bbafb914cbab9363f7aa13fedd2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5a7d825a62bdd5f9530132156a5cd17e"
server_key_iv: "2bc4d8f40bdf78e5f74db82d735e2dad"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.192474 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "35ca8bbafb914cbab9363f7aa13fedd2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5a7d825a62bdd5f9530132156a5cd17e"
server_key_iv: "2bc4d8f40bdf78e5f74db82d735e2dad"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.193794 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:33.194518 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.194691 22555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:33.194790 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "35ca8bbafb914cbab9363f7aa13fedd2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5a7d825a62bdd5f9530132156a5cd17e"
server_key_iv: "2bc4d8f40bdf78e5f74db82d735e2dad"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.194880 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:33.217942 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:33.220531 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:33.220705 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:33.220960 22555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:33.221300 22555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:33.221359 22555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.221395 22555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:33.221443 22555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.227304 22555 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:42155
I20260925 07:24:33.227393 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:42155 every 8 connection(s)
I20260925 07:24:33.227726 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:24:33 dist-test-slave-zw07 krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:33.236480 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22555
I20260925 07:24:33.236613 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:33.236876 4284 external_mini_cluster.cc:1428] Setting key 7057a8704897ffd3792b183f4076fb54
I20260925 07:24:33.238075 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.228542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:41505 (local address 127.4.47.62:40601)
0925 07:24:33.228611 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.228613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.228838 (+ 225us) server_negotiation.cc:407] Connection header received
0925 07:24:33.229395 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.229396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.229406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.229460 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:33.230448 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.230958 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.232116 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.232310 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.232370 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.234649 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:33.234651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:33.234652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:33.234659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:33.236134 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.236706 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.236707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.236708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.236774 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.237083 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.237084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.237085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.237247 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:33.237300 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.237931 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.238008 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:24:33.238382 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.228527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40601 (local address 127.4.47.2:41505)
0925 07:24:33.228697 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.228740 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.229302 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.229519 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.229522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.230145 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:33.230368 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.230373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.231044 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.231044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.232032 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.232034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.232398 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.232399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.232515 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.233025 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:33.233034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:33.234549 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:33.236234 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.236245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.236247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.236618 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.236845 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.236846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.236846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.236998 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.237552 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:33.237559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:33.237755 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.237896 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.238088 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260925 07:24:33.239005 22684 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40601
I20260925 07:24:33.239136 22684 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:33.239359 22684 heartbeater.cc:511] Master 127.4.47.62:40601 requested a full tablet report, sending...
I20260925 07:24:33.239770 22359 ts_manager.cc:194] Registered new tserver with Master: 35ca8bbafb914cbab9363f7aa13fedd2 (127.4.47.2:42155)
I20260925 07:24:33.240200 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:41505
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:24:33.263811 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:33.346380 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:33.346676 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:33.346720 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:33.348022 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:33.348096 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:33.348145 22689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:33.349668 22689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:40601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:33.349963 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:33.350208 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:33.355556 22695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.355551 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.355542 22697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:33.355752 22689 server_base.cc:1034] running on GCE node
I20260925 07:24:33.356060 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:33.356355 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:33.357501 22689 hybrid_clock.cc:648] HybridClock initialized: now 1790321073357492 us; error 26 us; skew 500 ppm
Sep 25 07:24:33 dist-test-slave-zw07 krb5kdc[22314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:33.360294 22689 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:24:33.360695 22689 webserver.cc:460] Webserver started at http://127.4.47.3:37913/ using document root <none> and password file <none>
I20260925 07:24:33.360891 22689 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:33.360932 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:33.361022 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:33.362043 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "001a683ac0b3463581a92e6b24d004b2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "59524c35224354fe72eb794da3097a9a"
server_key_iv: "12f78ea30494e3d02b3025621443d80d"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.362356 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "001a683ac0b3463581a92e6b24d004b2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "59524c35224354fe72eb794da3097a9a"
server_key_iv: "12f78ea30494e3d02b3025621443d80d"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.363780 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 07:24:33.364470 22704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.364639 22689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:33.364763 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "001a683ac0b3463581a92e6b24d004b2"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "59524c35224354fe72eb794da3097a9a"
server_key_iv: "12f78ea30494e3d02b3025621443d80d"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.364858 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:33.372550 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:33.374413 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:33.374594 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:33.374845 22689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:33.375209 22689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:33.375268 22689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.375312 22689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:33.375339 22689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.381302 22689 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:33295
I20260925 07:24:33.381350 22817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:33295 every 8 connection(s)
I20260925 07:24:33.381762 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:24:33 dist-test-slave-zw07 krb5kdc[22314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321073, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:33.388569 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22689
I20260925 07:24:33.388695 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:33.388890 4284 external_mini_cluster.cc:1428] Setting key 7378661f08697ed458c15367892350b0
I20260925 07:24:33.391342 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.382518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:52851 (local address 127.4.47.62:40601)
0925 07:24:33.382626 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.382629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.382795 (+ 166us) server_negotiation.cc:407] Connection header received
0925 07:24:33.383406 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.383407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.383421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.383471 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:33.384456 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.384973 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.385986 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.386202 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.386292 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.388412 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:33.388414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:33.388414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:33.388422 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:33.390096 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.390475 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.390475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.390476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.390524 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.390742 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.390743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.390743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.390861 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:33.390909 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.391212 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.391277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:24:33.391549 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.382474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:40601 (local address 127.4.47.3:52851)
0925 07:24:33.382656 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.382707 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.383288 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.383546 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.383554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.384137 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:33.384342 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.384347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.385074 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.385075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.385892 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.385894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.386297 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.386298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.386424 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.386894 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:33.386902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:33.388295 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:33.390165 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.390171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.390173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.390408 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.390579 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:33.390580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:33.390580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:33.390679 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:33.390915 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:33.390919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:33.391040 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.391193 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.391319 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 07:24:33.392026 22818 heartbeater.cc:347] Connected to a master server at 127.4.47.62:40601
I20260925 07:24:33.392130 22818 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:33.392310 22818 heartbeater.cc:511] Master 127.4.47.62:40601 requested a full tablet report, sending...
I20260925 07:24:33.392699 22359 ts_manager.cc:194] Registered new tserver with Master: 001a683ac0b3463581a92e6b24d004b2 (127.4.47.3:33295)
I20260925 07:24:33.393075 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:52851
I20260925 07:24:33.400861 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:33.408782 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.402478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.4.47.62:40601)
0925 07:24:33.402530 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.402532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.402592 (+ 60us) server_negotiation.cc:407] Connection header received
0925 07:24:33.402673 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.402674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.402683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.402742 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:33.403383 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.403895 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.404615 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.404819 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.404903 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.406208 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:33.406209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:33.406209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:33.406218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:33.407473 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.407875 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.407877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.407877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.407934 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:33.408171 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:33.408172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:33.408172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:33.408330 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:33.408451 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.408588 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.408675 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:24:33.410542 22359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:33.412534 22359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:33.419055 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.414892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34303 (local address 127.0.0.1:55218)
0925 07:24:33.415060 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.415131 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.415201 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.415456 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.415459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.415461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:33.415544 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.415546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.416672 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.416674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.417742 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.417746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.418682 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.418683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.418808 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.418824 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.418905 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.418960 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260925 07:24:33.419108 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.415090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.4.47.1:34303)
0925 07:24:33.415286 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.415292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.415304 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:24:33.415347 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.415350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.415367 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.415452 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:33.415623 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.416567 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.417845 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.418464 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.418533 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.418594 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.418937 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.418985 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260925 07:24:33.419641 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.414899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:33295 (local address 127.0.0.1:54684)
0925 07:24:33.415313 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.415322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.415372 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.416114 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.416115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.416117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:33.416160 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.416162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.417247 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.417248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.418356 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.418359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.419397 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.419398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.419451 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.419455 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.419528 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.419571 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":8,"thread_start_us":88,"threads_started":1}
I20260925 07:24:33.419667 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.415156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.4.47.3:33295)
0925 07:24:33.415908 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.415912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.415920 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:33.415955 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.415957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.415970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.416039 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:33.416235 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.417128 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.418450 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.419241 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.419306 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.419368 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.419546 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.419585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":92,"threads_started":1}
I20260925 07:24:33.420040 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.414898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:42155 (local address 127.0.0.1:38384)
0925 07:24:33.415266 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.415278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.415334 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.416662 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.416663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.416665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:33.416701 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.416703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.417794 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.417795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.418758 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.418760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.419779 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.419781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.419861 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.419865 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.419920 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.419959 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":89,"threads_started":1}
I20260925 07:24:33.420099 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.415081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.4.47.2:42155)
0925 07:24:33.415597 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.415601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.415612 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:24:33.415650 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.415652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.415668 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.415775 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:33.416782 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.417672 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.418852 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.419599 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.419681 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.419739 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.419958 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.420003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":87,"threads_started":1}
I20260925 07:24:33.420245 22484 tablet_service.cc:1467] Processing CreateTablet for tablet a1116ada15a941adb5af48ce7c7ef074 (DEFAULT_TABLE table=test-table [id=d398265277984554aefed1fcf648d672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:33.420632 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116ada15a941adb5af48ce7c7ef074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:33.420794 22618 tablet_service.cc:1467] Processing CreateTablet for tablet a1116ada15a941adb5af48ce7c7ef074 (DEFAULT_TABLE table=test-table [id=d398265277984554aefed1fcf648d672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:33.421098 22618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116ada15a941adb5af48ce7c7ef074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:33.424324 22835 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Bootstrap starting.
I20260925 07:24:33.424445 22836 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Bootstrap starting.
I20260925 07:24:33.424763 22752 tablet_service.cc:1467] Processing CreateTablet for tablet a1116ada15a941adb5af48ce7c7ef074 (DEFAULT_TABLE table=test-table [id=d398265277984554aefed1fcf648d672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:33.425084 22752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1116ada15a941adb5af48ce7c7ef074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:33.425096 22835 tablet_bootstrap.cc:654] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:33.425096 22836 tablet_bootstrap.cc:654] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:33.425755 22835 log.cc:824] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:33.425762 22836 log.cc:824] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:33.426560 22835 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: No bootstrap required, opened a new log
I20260925 07:24:33.426631 22835 ts_tablet_manager.cc:1397] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:33.426806 22836 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: No bootstrap required, opened a new log
I20260925 07:24:33.426880 22836 ts_tablet_manager.cc:1397] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:33.428037 22839 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Bootstrap starting.
I20260925 07:24:33.428640 22839 tablet_bootstrap.cc:654] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:33.428917 22836 raft_consensus.cc:348] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.428931 22835 raft_consensus.cc:348] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.429108 22835 raft_consensus.cc:374] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:33.429108 22836 raft_consensus.cc:374] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:33.429145 22836 raft_consensus.cc:729] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ca8bbafb914cbab9363f7aa13fedd2, State: Initialized, Role: FOLLOWER
I20260925 07:24:33.429145 22835 raft_consensus.cc:729] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5261261250d1478d9af1903baeb29d94, State: Initialized, Role: FOLLOWER
I20260925 07:24:33.429179 22839 log.cc:824] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:33.429291 22836 consensus_queue.cc:260] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.429291 22835 consensus_queue.cc:260] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.429581 22836 ts_tablet_manager.cc:1428] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:33.429581 22835 ts_tablet_manager.cc:1428] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:24:33.429601 22684 heartbeater.cc:503] Master 127.4.47.62:40601 was elected leader, sending a full tablet report...
I20260925 07:24:33.429749 22550 heartbeater.cc:503] Master 127.4.47.62:40601 was elected leader, sending a full tablet report...
I20260925 07:24:33.430279 22839 tablet_bootstrap.cc:492] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: No bootstrap required, opened a new log
I20260925 07:24:33.430344 22839 ts_tablet_manager.cc:1397] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:24:33.432049 22839 raft_consensus.cc:348] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.432199 22839 raft_consensus.cc:374] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:33.432224 22839 raft_consensus.cc:729] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001a683ac0b3463581a92e6b24d004b2, State: Initialized, Role: FOLLOWER
I20260925 07:24:33.432367 22839 consensus_queue.cc:260] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.432646 22839 ts_tablet_manager.cc:1428] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:24:33.432674 22818 heartbeater.cc:503] Master 127.4.47.62:40601 was elected leader, sending a full tablet report...
W20260925 07:24:33.478309 22685 tablet.cc:2369] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:33.554064 22551 tablet.cc:2369] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:33.632404 22819 tablet.cc:2369] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:33.652187 22843 raft_consensus.cc:482] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:33.652315 22843 raft_consensus.cc:504] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.652700 22843 leader_election.cc:288] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ca8bbafb914cbab9363f7aa13fedd2 (127.4.47.2:42155), 5261261250d1478d9af1903baeb29d94 (127.4.47.1:34303)
I20260925 07:24:33.656690 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.652991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:50091 (local address 127.4.47.2:42155)
0925 07:24:33.653041 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.653043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.653055 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:24:33.653165 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.653167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.653182 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.653240 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:33.653433 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.654184 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.655695 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.656281 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.656371 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.656379 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.656572 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.656613 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:24:33.656690 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.653110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:37869 (local address 127.4.47.1:34303)
0925 07:24:33.653182 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.653185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.653319 (+ 134us) server_negotiation.cc:407] Connection header received
0925 07:24:33.653368 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.653369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.653379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.653452 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:33.653604 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.654299 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.655694 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.656296 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.656366 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.656379 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.656572 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.656613 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:24:33.656723 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.652925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:42155 (local address 127.4.47.3:50091)
0925 07:24:33.652985 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.652999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.653051 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.653274 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.653275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.653278 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:33.653357 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.653359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.654277 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.654277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.655592 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.655595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.656394 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.656395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.656487 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.656488 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.656553 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.656609 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":12}
I20260925 07:24:33.656741 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.652987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34303 (local address 127.4.47.3:37869)
0925 07:24:33.653209 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.653242 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:33.653308 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:33.653476 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:33.653477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:33.653478 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:33.653534 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.653535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.654384 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.654385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.655592 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:33.655596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.656394 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.656395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:33.656482 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.656487 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:24:33.656553 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 07:24:33.656609 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":11,"thread_start_us":139,"threads_started":1}
I20260925 07:24:33.657140 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116ada15a941adb5af48ce7c7ef074" candidate_uuid: "001a683ac0b3463581a92e6b24d004b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" is_pre_election: true
I20260925 07:24:33.657140 22504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116ada15a941adb5af48ce7c7ef074" candidate_uuid: "001a683ac0b3463581a92e6b24d004b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5261261250d1478d9af1903baeb29d94" is_pre_election: true
I20260925 07:24:33.657325 22638 raft_consensus.cc:2454] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001a683ac0b3463581a92e6b24d004b2 in term 0.
I20260925 07:24:33.657325 22504 raft_consensus.cc:2454] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001a683ac0b3463581a92e6b24d004b2 in term 0.
I20260925 07:24:33.657599 22706 leader_election.cc:302] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001a683ac0b3463581a92e6b24d004b2, 35ca8bbafb914cbab9363f7aa13fedd2; no voters:
I20260925 07:24:33.657730 22843 raft_consensus.cc:2781] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:33.657786 22843 raft_consensus.cc:482] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:33.657814 22843 raft_consensus.cc:3037] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:33.658527 22843 raft_consensus.cc:504] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.658679 22843 leader_election.cc:288] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [CANDIDATE]: Term 1 election: Requested vote from peers 35ca8bbafb914cbab9363f7aa13fedd2 (127.4.47.2:42155), 5261261250d1478d9af1903baeb29d94 (127.4.47.1:34303)
I20260925 07:24:33.658850 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116ada15a941adb5af48ce7c7ef074" candidate_uuid: "001a683ac0b3463581a92e6b24d004b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ca8bbafb914cbab9363f7aa13fedd2"
I20260925 07:24:33.658926 22638 raft_consensus.cc:3037] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:33.658903 22504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1116ada15a941adb5af48ce7c7ef074" candidate_uuid: "001a683ac0b3463581a92e6b24d004b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5261261250d1478d9af1903baeb29d94"
I20260925 07:24:33.659013 22504 raft_consensus.cc:3037] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:33.659678 22638 raft_consensus.cc:2454] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001a683ac0b3463581a92e6b24d004b2 in term 1.
I20260925 07:24:33.659824 22504 raft_consensus.cc:2454] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001a683ac0b3463581a92e6b24d004b2 in term 1.
I20260925 07:24:33.659857 22706 leader_election.cc:302] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001a683ac0b3463581a92e6b24d004b2, 35ca8bbafb914cbab9363f7aa13fedd2; no voters:
I20260925 07:24:33.660024 22843 raft_consensus.cc:2781] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:33.660287 22843 raft_consensus.cc:686] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 1 LEADER]: Becoming Leader. State: Replica: 001a683ac0b3463581a92e6b24d004b2, State: Running, Role: LEADER
I20260925 07:24:33.660395 22843 consensus_queue.cc:237] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } }
I20260925 07:24:33.661222 22358 catalog_manager.cc:5495] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001a683ac0b3463581a92e6b24d004b2 (127.4.47.3). New cstate: current_term: 1 leader_uuid: "001a683ac0b3463581a92e6b24d004b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001a683ac0b3463581a92e6b24d004b2" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 33295 } health_report { overall_health: HEALTHY } } }
I20260925 07:24:33.703797 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.701085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.4.47.3:33295)
0925 07:24:33.701149 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.701151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.701160 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:24:33.701215 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.701216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.701240 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.701318 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:33.701450 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.701986 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.702791 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.702994 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.703124 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.703267 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:33.703358 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:33.703437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.703700 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.703710 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:24:33.705884 22504 raft_consensus.cc:1261] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 1 FOLLOWER]: Refusing update from remote peer 001a683ac0b3463581a92e6b24d004b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:33.705924 22638 raft_consensus.cc:1261] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 1 FOLLOWER]: Refusing update from remote peer 001a683ac0b3463581a92e6b24d004b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:33.706275 22843 consensus_queue.cc:1035] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5261261250d1478d9af1903baeb29d94" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:33.706486 22845 consensus_queue.cc:1035] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ca8bbafb914cbab9363f7aa13fedd2" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:33.708815 22848 mvcc.cc:205] Tried to move back new op lower bound from 7333155117896359936 to 7333155117713571840. Current Snapshot: MvccSnapshot[applied={T|T < 7333155117896359936}]
I20260925 07:24:33.711788 22359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:33.711911 22359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:33.712739 22359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 542c46be06414aaab689966d8b0dfa0b: Sending DeleteTablet for 3 replicas of tablet a1116ada15a941adb5af48ce7c7ef074
I20260925 07:24:33.713151 22484 tablet_service.cc:1514] Processing DeleteTablet for tablet a1116ada15a941adb5af48ce7c7ef074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260925 07:24:33.713147 22618 tablet_service.cc:1514] Processing DeleteTablet for tablet a1116ada15a941adb5af48ce7c7ef074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:38384
I20260925 07:24:33.713438 22858 tablet_replica.cc:330] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: stopping tablet replica
I20260925 07:24:33.713560 22858 raft_consensus.cc:2229] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:33.713686 22858 raft_consensus.cc:2258] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:33.713677 22752 tablet_service.cc:1514] Processing DeleteTablet for tablet a1116ada15a941adb5af48ce7c7ef074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:33 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:54684
I20260925 07:24:33.713897 22857 tablet_replica.cc:330] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: stopping tablet replica
I20260925 07:24:33.714057 22857 raft_consensus.cc:2229] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:33.714212 22857 raft_consensus.cc:2258] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:33.714288 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22421
I20260925 07:24:33.714208 22858 ts_tablet_manager.cc:1905] T a1116ada15a941adb5af48ce7c7ef074 P 5261261250d1478d9af1903baeb29d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:33.715476 22857 ts_tablet_manager.cc:1905] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:33.716221 22859 tablet_replica.cc:330] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: stopping tablet replica
I20260925 07:24:33.716382 22859 raft_consensus.cc:2229] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:33.716524 22859 raft_consensus.cc:2258] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:33.716977 22859 ts_tablet_manager.cc:1905] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:33.717501 22857 ts_tablet_manager.cc:1918] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:33.717635 22857 log.cc:1196] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/a1116ada15a941adb5af48ce7c7ef074
I20260925 07:24:33.717994 22857 ts_tablet_manager.cc:1939] T a1116ada15a941adb5af48ce7c7ef074 P 35ca8bbafb914cbab9363f7aa13fedd2: Deleting consensus metadata
I20260925 07:24:33.718536 22344 catalog_manager.cc:4841] TS 35ca8bbafb914cbab9363f7aa13fedd2 (127.4.47.2:42155): tablet a1116ada15a941adb5af48ce7c7ef074 (table test-table [id=d398265277984554aefed1fcf648d672]) successfully deleted
I20260925 07:24:33.718992 22859 ts_tablet_manager.cc:1918] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:33.719098 22859 log.cc:1196] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/a1116ada15a941adb5af48ce7c7ef074
I20260925 07:24:33.719395 22859 ts_tablet_manager.cc:1939] T a1116ada15a941adb5af48ce7c7ef074 P 001a683ac0b3463581a92e6b24d004b2: Deleting consensus metadata
I20260925 07:24:33.719806 22346 catalog_manager.cc:4841] TS 001a683ac0b3463581a92e6b24d004b2 (127.4.47.3:33295): tablet a1116ada15a941adb5af48ce7c7ef074 (table test-table [id=d398265277984554aefed1fcf648d672]) successfully deleted
W20260925 07:24:33.721293 22343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:34303 (error 108)
I20260925 07:24:33.721647 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22555
I20260925 07:24:33.722272 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.721745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:34303 (local address 127.0.0.1:55234)
0925 07:24:33.722173 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:33.722220 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:34303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20260925 07:24:33.722527 22343 catalog_manager.cc:4584] TS 5261261250d1478d9af1903baeb29d94 (127.4.47.1:34303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1116ada15a941adb5af48ce7c7ef074: Network error: Client connection negotiation failed: client connection to 127.4.47.1:34303: connect: Connection refused (error 111)
I20260925 07:24:33.726902 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22689
I20260925 07:24:33.732812 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22327
2026-09-25T07:24:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T07:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:33Z Disabled control of system clock
I20260925 07:24:33.760991 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38369
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:33.844066 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:33.844383 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:33.844408 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:33.845777 22864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:33.845826 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:33.845839 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:33.845852 22864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:33.845865 22864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:33.847343 22864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:33.847610 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:33.847937 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:33.853122 22870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.853108 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:33.853102 22869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:33.853139 22864 server_base.cc:1034] running on GCE node
I20260925 07:24:33.853550 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:33.853844 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:33.854996 22864 hybrid_clock.cc:648] HybridClock initialized: now 1790321073854981 us; error 49 us; skew 500 ppm
I20260925 07:24:33.861546 22864 webserver.cc:460] Webserver started at http://127.4.47.62:38753/ using document root <none> and password file <none>
I20260925 07:24:33.861806 22864 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:33.861856 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:33.861958 22864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:33.863164 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "3da4f8bb3c2448a28112270d16c5db51"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5d192974c571c3dd9b7267cd642d141a"
server_key_iv: "3e991de7b42c838c8d4f279097a5bd1f"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.863498 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "3da4f8bb3c2448a28112270d16c5db51"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5d192974c571c3dd9b7267cd642d141a"
server_key_iv: "3e991de7b42c838c8d4f279097a5bd1f"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.865069 22864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:33.866163 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:33.866412 22864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:33.866542 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "3da4f8bb3c2448a28112270d16c5db51"
format_stamp: "Formatted at 2026-09-25 07:24:33 on dist-test-slave-zw07"
server_key: "5d192974c571c3dd9b7267cd642d141a"
server_key_iv: "3e991de7b42c838c8d4f279097a5bd1f"
server_key_version: "encryptionkey@0"
I20260925 07:24:33.866636 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:33.879730 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:33.880024 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:33.880146 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:33.884447 22864 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:38369
I20260925 07:24:33.884544 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:38369 every 8 connection(s)
I20260925 07:24:33.884886 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:33.885859 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22864
I20260925 07:24:33.885998 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:33.886080 22931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:33.886314 4284 external_mini_cluster.cc:1428] Setting key 7733035eef5be9f7b1584de74e073e30
I20260925 07:24:33.889348 22931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: Bootstrap starting.
I20260925 07:24:33.890091 22931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:33.890450 22931 log.cc:824] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:33.891346 22931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: No bootstrap required, opened a new log
I20260925 07:24:33.892441 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:33.886925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.4.47.62:38369)
0925 07:24:33.887163 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:33.887172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:33.887210 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:24:33.887683 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:33.887690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:33.887754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:33.887871 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:33.888170 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.889074 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.890217 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:33.890776 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:33.890906 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:33.891756 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:33.891760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:33.891760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:33.891921 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:33.891953 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:33.891963 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:33.892016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:33.892117 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 07:24:33.892221 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260925 07:24:33.893504 22931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } }
I20260925 07:24:33.893648 22931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:33.893679 22931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da4f8bb3c2448a28112270d16c5db51, State: Initialized, Role: FOLLOWER
I20260925 07:24:33.893833 22931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } }
I20260925 07:24:33.893898 22931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:33.893919 22931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:33.893987 22931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:33.894626 22931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } }
I20260925 07:24:33.894747 22931 leader_election.cc:302] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da4f8bb3c2448a28112270d16c5db51; no voters:
I20260925 07:24:33.894959 22931 leader_election.cc:288] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:33.895107 22936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:33.895275 22936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [term 1 LEADER]: Becoming Leader. State: Replica: 3da4f8bb3c2448a28112270d16c5db51, State: Running, Role: LEADER
I20260925 07:24:33.895360 22931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:33.895385 22936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } }
I20260925 07:24:33.895869 22938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da4f8bb3c2448a28112270d16c5db51. Latest consensus state: current_term: 1 leader_uuid: "3da4f8bb3c2448a28112270d16c5db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } } }
I20260925 07:24:33.895886 22937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da4f8bb3c2448a28112270d16c5db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da4f8bb3c2448a28112270d16c5db51" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38369 } } }
I20260925 07:24:33.895967 22938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:33.896015 22937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:33.896258 22945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:33.896800 22945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:33.898377 22945 catalog_manager.cc:1347] Generated new cluster ID: a02f6bb2ecc94b2098570c0b4708f6d5
I20260925 07:24:33.898430 22945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:33.904404 22945 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:33.904963 22945 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:33.916675 22945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: Generated new TSK 0
I20260925 07:24:33.916905 22945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:33.925427 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--builtin_ntp_servers=127.4.47.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:34.010341 22955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:34.010610 22955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:34.010649 22955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:34.011981 22955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:34.012056 22955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:34.012104 22955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:34.013551 22955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:34.013849 22955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:34.014171 22955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:34.019786 22963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:34.019774 22961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:34.019774 22960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:34.020176 22955 server_base.cc:1034] running on GCE node
I20260925 07:24:34.020454 22955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:34.020756 22955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:34.021936 22955 hybrid_clock.cc:648] HybridClock initialized: now 1790321074021920 us; error 31 us; skew 500 ppm
I20260925 07:24:34.023494 22955 webserver.cc:460] Webserver started at http://127.4.47.1:40951/ using document root <none> and password file <none>
I20260925 07:24:34.023792 22955 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:34.023865 22955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:34.024011 22955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:34.025128 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "2bcaf0a18b01483bb21c6edb93523299"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "9b9382e875a44691fb326cb30eb1b0f5"
server_key_iv: "eb5829e636200ab30bf722ecddcc1433"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.025511 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "2bcaf0a18b01483bb21c6edb93523299"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "9b9382e875a44691fb326cb30eb1b0f5"
server_key_iv: "eb5829e636200ab30bf722ecddcc1433"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.026916 22955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:34.027837 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.028025 22955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:34.028138 22955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "2bcaf0a18b01483bb21c6edb93523299"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "9b9382e875a44691fb326cb30eb1b0f5"
server_key_iv: "eb5829e636200ab30bf722ecddcc1433"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.028223 22955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:34.041590 22955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:34.041939 22955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:34.042102 22955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:34.042351 22955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:34.042695 22955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:34.042749 22955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.042794 22955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:34.042819 22955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.049084 22955 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:37397
I20260925 07:24:34.049139 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:37397 every 8 connection(s)
I20260925 07:24:34.049539 22955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:24:34.050926 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 22955
I20260925 07:24:34.051028 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:34.051239 4284 external_mini_cluster.cc:1428] Setting key b1b9a8c25f8e6cbbd1184699249b9adf
I20260925 07:24:34.052336 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--builtin_ntp_servers=127.4.47.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:24:34.056344 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.050387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:44781 (local address 127.4.47.62:38369)
0925 07:24:34.050439 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.050441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.050668 (+ 227us) server_negotiation.cc:407] Connection header received
0925 07:24:34.051431 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.051432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.051450 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.051533 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:34.052482 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.053287 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.054574 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.054922 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.055000 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.056032 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:34.056034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:34.056034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:34.056038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:34.056052 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:34.056055 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:34.056096 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.056182 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.056211 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:24:34.056607 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.050322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38369 (local address 127.4.47.1:44781)
0925 07:24:34.050529 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.050572 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.051311 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.051615 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.051619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.051630 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:34.052359 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.052366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.053424 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.053425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.054481 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.054483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.055042 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.055043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.055244 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.055870 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:34.055870 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:34.055912 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:34.055943 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:34.056106 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:34.056111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:34.056114 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.056178 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.056304 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260925 07:24:34.057183 23083 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38369
I20260925 07:24:34.057303 23083 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:34.057512 23083 heartbeater.cc:511] Master 127.4.47.62:38369 requested a full tablet report, sending...
I20260925 07:24:34.058082 22895 ts_manager.cc:194] Registered new tserver with Master: 2bcaf0a18b01483bb21c6edb93523299 (127.4.47.1:37397)
I20260925 07:24:34.058704 22895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.1:44781
W20260925 07:24:34.135320 23086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:34.135535 23086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:34.135558 23086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:34.136777 23086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:34.136826 23086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:34.136854 23086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:34.138312 23086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:34.138595 23086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:34.138923 23086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:34.144129 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:34.144119 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:34.144145 23086 server_base.cc:1034] running on GCE node
W20260925 07:24:34.144119 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:34.144582 23086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:34.144852 23086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:34.146032 23086 hybrid_clock.cc:648] HybridClock initialized: now 1790321074146012 us; error 37 us; skew 500 ppm
I20260925 07:24:34.147715 23086 webserver.cc:460] Webserver started at http://127.4.47.2:39897/ using document root <none> and password file <none>
I20260925 07:24:34.148011 23086 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:34.148087 23086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:34.148253 23086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:34.149528 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "fea5cf672e2046fc8ca5553686639201"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "34df001ea09c4fa8b4232def7737ed4d"
server_key_iv: "49f9084598faaae52790bbda361ba92f"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.150053 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "fea5cf672e2046fc8ca5553686639201"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "34df001ea09c4fa8b4232def7737ed4d"
server_key_iv: "49f9084598faaae52790bbda361ba92f"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.151580 23086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 07:24:34.152415 23100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.152581 23086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:34.152686 23086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "fea5cf672e2046fc8ca5553686639201"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "34df001ea09c4fa8b4232def7737ed4d"
server_key_iv: "49f9084598faaae52790bbda361ba92f"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.152776 23086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:34.163617 23086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:34.163971 23086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:34.164135 23086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:34.164384 23086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:34.164736 23086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:34.164793 23086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.164834 23086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:34.164858 23086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.170775 23086 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:35373
I20260925 07:24:34.170855 23213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:35373 every 8 connection(s)
I20260925 07:24:34.171268 23086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:24:34.176750 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.172097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55719 (local address 127.4.47.62:38369)
0925 07:24:34.172153 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.172156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.172453 (+ 297us) server_negotiation.cc:407] Connection header received
0925 07:24:34.172969 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.172970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.172980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.173035 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:34.173702 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.174225 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.175291 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.175489 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.175547 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.176520 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:34.176521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:34.176521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:34.176526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:34.176540 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:34.176543 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:34.176587 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.176655 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.176685 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:24:34.176999 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.172107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38369 (local address 127.4.47.2:55719)
0925 07:24:34.172316 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.172370 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.172883 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.173090 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.173093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.173101 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:34.173616 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.173619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.174314 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.174315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.175208 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.175210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.175589 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.175590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.175720 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.176378 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:34.176378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:34.176407 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:34.176431 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:34.176589 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:34.176591 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:34.176594 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.176643 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.176738 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260925 07:24:34.177623 23214 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38369
I20260925 07:24:34.177769 23214 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:34.177969 23214 heartbeater.cc:511] Master 127.4.47.62:38369 requested a full tablet report, sending...
I20260925 07:24:34.178324 22895 ts_manager.cc:194] Registered new tserver with Master: fea5cf672e2046fc8ca5553686639201 (127.4.47.2:35373)
I20260925 07:24:34.178428 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23086
I20260925 07:24:34.178539 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:34.178735 4284 external_mini_cluster.cc:1428] Setting key 1ef52a348ab665829e0907c55d1dc767
I20260925 07:24:34.178785 22895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.2:55719
I20260925 07:24:34.179669 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--builtin_ntp_servers=127.4.47.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:34.264710 23217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:34.264936 23217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:34.264962 23217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:34.266170 23217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:34.266214 23217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:34.266242 23217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:34.267740 23217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:34.268016 23217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:34.268333 23217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:24:34.273696 23217 server_base.cc:1034] running on GCE node
W20260925 07:24:34.273663 23222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:34.273651 23223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:34.273661 23225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:34.274148 23217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:34.274370 23217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:34.275506 23217 hybrid_clock.cc:648] HybridClock initialized: now 1790321074275481 us; error 41 us; skew 500 ppm
I20260925 07:24:34.276731 23217 webserver.cc:460] Webserver started at http://127.4.47.3:41577/ using document root <none> and password file <none>
I20260925 07:24:34.276983 23217 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:34.277029 23217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:34.277184 23217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:34.278328 23217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "d41a573d9371423987ba7be1de02bccc"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "bb973e5665272300e72687455e3138e4"
server_key_iv: "0556918f1bbf33aca0a9aabc0c75cc10"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.278697 23217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "d41a573d9371423987ba7be1de02bccc"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "bb973e5665272300e72687455e3138e4"
server_key_iv: "0556918f1bbf33aca0a9aabc0c75cc10"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.280154 23217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:34.280936 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.281104 23217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:24:34.281204 23217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "d41a573d9371423987ba7be1de02bccc"
format_stamp: "Formatted at 2026-09-25 07:24:34 on dist-test-slave-zw07"
server_key: "bb973e5665272300e72687455e3138e4"
server_key_iv: "0556918f1bbf33aca0a9aabc0c75cc10"
server_key_version: "encryptionkey@0"
I20260925 07:24:34.281288 23217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:34.298650 23217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:34.298985 23217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:34.299198 23217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:34.299465 23217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:34.299816 23217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:34.299873 23217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.299908 23217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:34.299937 23217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:34.306064 23217 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:40953
I20260925 07:24:34.306107 23344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:40953 every 8 connection(s)
I20260925 07:24:34.306495 23217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:34.311970 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.307286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:59797 (local address 127.4.47.62:38369)
0925 07:24:34.307340 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.307343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.307570 (+ 227us) server_negotiation.cc:407] Connection header received
0925 07:24:34.308117 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.308119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.308131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.308191 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:34.308913 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.309446 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.310487 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.310702 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.310796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.311661 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:34.311663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:34.311663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:34.311668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:34.311684 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:34.311689 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:34.311741 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.311859 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.311886 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:24:34.312201 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.307239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38369 (local address 127.4.47.3:59797)
0925 07:24:34.307434 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.307474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.308004 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.308262 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.308266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.308276 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:34.308801 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.308805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.309562 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.309563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.310385 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.310387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.310811 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.310812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.310958 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.311503 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:34.311504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:34.311533 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:34.311561 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:34.311755 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:34.311760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:34.311769 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.311841 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.311950 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260925 07:24:34.312740 23345 heartbeater.cc:347] Connected to a master server at 127.4.47.62:38369
I20260925 07:24:34.312846 23345 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:34.313073 23345 heartbeater.cc:511] Master 127.4.47.62:38369 requested a full tablet report, sending...
I20260925 07:24:34.313598 22895 ts_manager.cc:194] Registered new tserver with Master: d41a573d9371423987ba7be1de02bccc (127.4.47.3:40953)
I20260925 07:24:34.313956 22895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.3:59797
I20260925 07:24:34.315353 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23217
I20260925 07:24:34.315439 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:34.315585 4284 external_mini_cluster.cc:1428] Setting key 91bd147c4f0d092acd0cad6f741b12ce
I20260925 07:24:34.316608 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:34.320753 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.317928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.4.47.62:38369)
0925 07:24:34.317989 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.317991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.318072 (+ 81us) server_negotiation.cc:407] Connection header received
0925 07:24:34.318133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.318133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.318142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.318194 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:34.318342 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.318812 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.319594 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.319782 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.319835 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.320502 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:34.320503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:34.320504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:34.320507 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:34.320521 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:34.320524 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:34.320599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.320672 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.320694 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:24:34.322491 22895 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:34.324123 22895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:34.330787 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.326383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37397 (local address 127.0.0.1:58742)
0925 07:24:34.326589 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.326627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.326696 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.327029 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.327032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.327034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:34.327133 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.327136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328183 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.328184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.329414 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.329418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.330471 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.330472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.330552 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.330565 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.330631 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.330694 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260925 07:24:34.331125 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.326647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.4.47.1:37397)
0925 07:24:34.326846 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.326851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.326859 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:34.326893 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.326894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.326906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.327001 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:34.327240 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328085 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.329526 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.330272 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.330343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.330436 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.330962 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.331007 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260925 07:24:34.331241 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.326388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:40953 (local address 127.0.0.1:59676)
0925 07:24:34.326586 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.326626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.326695 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.327438 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.327439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.327441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:34.327476 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.327477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328781 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.328782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.329986 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.329989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.330977 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.330978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.331033 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.331038 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.331103 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.331149 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":41,"thread_start_us":112,"threads_started":1}
I20260925 07:24:34.331427 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.326383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:35373 (local address 127.0.0.1:57136)
0925 07:24:34.326586 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.326626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.326695 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.327334 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.327335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.327338 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:34.327575 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.327578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328913 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.328913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.329990 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.329992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.331007 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.331008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.331101 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.331105 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.331146 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.331261 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260925 07:24:34.331713 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.327033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.4.47.3:40953)
0925 07:24:34.327241 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.327246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.327256 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:24:34.327293 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.327295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.327310 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.327439 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:34.327555 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328681 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.330085 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.330866 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.331423 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.331520 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.331550 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.331616 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260925 07:24:34.331908 23017 tablet_service.cc:1467] Processing CreateTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 (DEFAULT_TABLE table=test-table [id=c1d055277618404eb6079a31a47e5a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:34.332304 23017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c0d3d48f444c1092126b3bbdde55f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:34.332406 23279 tablet_service.cc:1467] Processing CreateTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 (DEFAULT_TABLE table=test-table [id=c1d055277618404eb6079a31a47e5a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:34.332777 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c0d3d48f444c1092126b3bbdde55f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:34.335168 23360 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Bootstrap starting.
I20260925 07:24:34.335139 23359 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Bootstrap starting.
I20260925 07:24:34.335700 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.326855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.4.47.2:35373)
0925 07:24:34.327038 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.327043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.327052 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:24:34.327119 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.327120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.327134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.327669 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:34.327730 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.328819 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.330144 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.330816 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.331273 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.335486 (+ 4213us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.335527 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.335595 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260925 07:24:34.336054 23360 tablet_bootstrap.cc:654] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:34.336274 23359 tablet_bootstrap.cc:654] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:34.336378 23360 log.cc:824] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:34.336454 23148 tablet_service.cc:1467] Processing CreateTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 (DEFAULT_TABLE table=test-table [id=c1d055277618404eb6079a31a47e5a4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:34.336642 23359 log.cc:824] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:34.336825 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c0d3d48f444c1092126b3bbdde55f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:34.337205 23360 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: No bootstrap required, opened a new log
I20260925 07:24:34.337280 23360 ts_tablet_manager.cc:1397] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:24:34.337453 23359 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: No bootstrap required, opened a new log
I20260925 07:24:34.337538 23359 ts_tablet_manager.cc:1397] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:24:34.339694 23359 raft_consensus.cc:348] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.339870 23359 raft_consensus.cc:374] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:34.339915 23359 raft_consensus.cc:729] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcaf0a18b01483bb21c6edb93523299, State: Initialized, Role: FOLLOWER
I20260925 07:24:34.339937 23360 raft_consensus.cc:348] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.340097 23360 raft_consensus.cc:374] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:34.340054 23359 consensus_queue.cc:260] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.340163 23360 raft_consensus.cc:729] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41a573d9371423987ba7be1de02bccc, State: Initialized, Role: FOLLOWER
I20260925 07:24:34.340319 23360 consensus_queue.cc:260] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.340400 23359 ts_tablet_manager.cc:1428] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:24:34.340538 23083 heartbeater.cc:503] Master 127.4.47.62:38369 was elected leader, sending a full tablet report...
I20260925 07:24:34.340579 23360 ts_tablet_manager.cc:1428] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 07:24:34.340720 23345 heartbeater.cc:503] Master 127.4.47.62:38369 was elected leader, sending a full tablet report...
I20260925 07:24:34.340761 23363 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Bootstrap starting.
I20260925 07:24:34.341524 23363 tablet_bootstrap.cc:654] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:34.341918 23363 log.cc:824] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:34.342651 23363 tablet_bootstrap.cc:492] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: No bootstrap required, opened a new log
I20260925 07:24:34.342710 23363 ts_tablet_manager.cc:1397] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:24:34.344904 23363 raft_consensus.cc:348] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.345048 23363 raft_consensus.cc:374] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:34.345072 23363 raft_consensus.cc:729] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea5cf672e2046fc8ca5553686639201, State: Initialized, Role: FOLLOWER
I20260925 07:24:34.345207 23363 consensus_queue.cc:260] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.345480 23363 ts_tablet_manager.cc:1428] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:24:34.345645 23214 heartbeater.cc:503] Master 127.4.47.62:38369 was elected leader, sending a full tablet report...
W20260925 07:24:34.421875 23215 tablet.cc:2369] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:34.550302 23084 tablet.cc:2369] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:34.557158 23346 tablet.cc:2369] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:34.599382 23365 raft_consensus.cc:482] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:34.599565 23365 raft_consensus.cc:504] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.599979 23365 leader_election.cc:288] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fea5cf672e2046fc8ca5553686639201 (127.4.47.2:35373), 2bcaf0a18b01483bb21c6edb93523299 (127.4.47.1:37397)
I20260925 07:24:34.602891 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.600198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37397 (local address 127.4.47.3:37605)
0925 07:24:34.600263 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.600276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.600349 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.600513 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.600514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.600515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:34.600591 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.600593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.601247 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.601248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.602028 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.602031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.602621 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.602699 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.602702 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.602766 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.602810 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 07:24:34.602892 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.600321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:37605 (local address 127.4.47.1:37397)
0925 07:24:34.600367 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.600369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.600379 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:24:34.600423 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.600423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.600441 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.600508 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:34.600671 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.601157 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602139 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.602529 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602620 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.602630 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.602785 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.602825 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:24:34.603276 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.600287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:49201 (local address 127.4.47.2:35373)
0925 07:24:34.600346 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.600348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.600506 (+ 158us) server_negotiation.cc:407] Connection header received
0925 07:24:34.600585 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.600586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.600595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.600667 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:34.600845 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.601322 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602422 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.602820 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602911 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.602923 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.603146 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.603192 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:24:34.603255 23037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c0d3d48f444c1092126b3bbdde55f4" candidate_uuid: "d41a573d9371423987ba7be1de02bccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcaf0a18b01483bb21c6edb93523299" is_pre_election: true
I20260925 07:24:34.603412 23037 raft_consensus.cc:2454] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41a573d9371423987ba7be1de02bccc in term 0.
I20260925 07:24:34.603531 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.600255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:35373 (local address 127.4.47.3:49201)
0925 07:24:34.600428 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.600435 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:34.600516 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:34.600682 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:34.600683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:34.600685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:34.600772 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.600774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.601409 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.601410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.602220 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:34.602222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.602925 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.602927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:34.603019 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.603024 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:24:34.603387 (+ 363us) client_negotiation.cc:240] Negotiation successful
0925 07:24:34.603439 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260925 07:24:34.603705 23233 leader_election.cc:302] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcaf0a18b01483bb21c6edb93523299, d41a573d9371423987ba7be1de02bccc; no voters:
I20260925 07:24:34.603881 23365 raft_consensus.cc:2781] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:34.603951 23365 raft_consensus.cc:482] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:34.603981 23365 raft_consensus.cc:3037] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:34.603950 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c0d3d48f444c1092126b3bbdde55f4" candidate_uuid: "d41a573d9371423987ba7be1de02bccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea5cf672e2046fc8ca5553686639201" is_pre_election: true
I20260925 07:24:34.604133 23168 raft_consensus.cc:2454] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41a573d9371423987ba7be1de02bccc in term 0.
I20260925 07:24:34.604794 23365 raft_consensus.cc:504] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.604962 23365 leader_election.cc:288] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [CANDIDATE]: Term 1 election: Requested vote from peers fea5cf672e2046fc8ca5553686639201 (127.4.47.2:35373), 2bcaf0a18b01483bb21c6edb93523299 (127.4.47.1:37397)
I20260925 07:24:34.605209 23037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c0d3d48f444c1092126b3bbdde55f4" candidate_uuid: "d41a573d9371423987ba7be1de02bccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcaf0a18b01483bb21c6edb93523299"
I20260925 07:24:34.605300 23037 raft_consensus.cc:3037] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:34.605981 23037 raft_consensus.cc:2454] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41a573d9371423987ba7be1de02bccc in term 1.
I20260925 07:24:34.606117 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c0d3d48f444c1092126b3bbdde55f4" candidate_uuid: "d41a573d9371423987ba7be1de02bccc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea5cf672e2046fc8ca5553686639201"
I20260925 07:24:34.606180 23233 leader_election.cc:302] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcaf0a18b01483bb21c6edb93523299, d41a573d9371423987ba7be1de02bccc; no voters:
I20260925 07:24:34.606211 23168 raft_consensus.cc:3037] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:34.606276 23365 raft_consensus.cc:2781] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:34.606462 23365 raft_consensus.cc:686] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 1 LEADER]: Becoming Leader. State: Replica: d41a573d9371423987ba7be1de02bccc, State: Running, Role: LEADER
I20260925 07:24:34.606565 23365 consensus_queue.cc:237] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } }
I20260925 07:24:34.607141 23168 raft_consensus.cc:2454] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41a573d9371423987ba7be1de02bccc in term 1.
I20260925 07:24:34.607455 22894 catalog_manager.cc:5495] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc reported cstate change: term changed from 0 to 1, leader changed from <none> to d41a573d9371423987ba7be1de02bccc (127.4.47.3). New cstate: current_term: 1 leader_uuid: "d41a573d9371423987ba7be1de02bccc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41a573d9371423987ba7be1de02bccc" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 40953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 } health_report { overall_health: UNKNOWN } } }
I20260925 07:24:34.615522 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.613161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.4.47.3:40953)
0925 07:24:34.613266 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:34.613268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:34.613284 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:24:34.613320 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:34.613321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:34.613344 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:34.613407 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:34.613547 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.614023 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.614743 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:34.614927 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:34.614979 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:34.615189 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:34.615292 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:34.615374 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:34.615430 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 07:24:34.615438 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 07:24:34.617882 23168 raft_consensus.cc:1261] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 1 FOLLOWER]: Refusing update from remote peer d41a573d9371423987ba7be1de02bccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:34.617884 23037 raft_consensus.cc:1261] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 1 FOLLOWER]: Refusing update from remote peer d41a573d9371423987ba7be1de02bccc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:34.618283 23369 consensus_queue.cc:1035] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea5cf672e2046fc8ca5553686639201" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 35373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:34.618445 23365 consensus_queue.cc:1035] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bcaf0a18b01483bb21c6edb93523299" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:34.623481 22894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:34.623585 22894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:34.624370 22894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3da4f8bb3c2448a28112270d16c5db51: Sending DeleteTablet for 3 replicas of tablet d0c0d3d48f444c1092126b3bbdde55f4
I20260925 07:24:34.624742 23279 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:34 UTC) from {username='slave'} at 127.0.0.1:59676
I20260925 07:24:34.624727 23017 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:34 UTC) from {username='slave'} at 127.0.0.1:58742
I20260925 07:24:34.624905 23148 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c0d3d48f444c1092126b3bbdde55f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:34 UTC) from {username='slave'} at 127.0.0.1:57136
I20260925 07:24:34.625221 23382 tablet_replica.cc:330] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299: stopping tablet replica
I20260925 07:24:34.625330 23382 raft_consensus.cc:2229] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:34.625164 23383 tablet_replica.cc:330] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: stopping tablet replica
I20260925 07:24:34.625440 23383 raft_consensus.cc:2229] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:34.625459 23382 raft_consensus.cc:2258] T d0c0d3d48f444c1092126b3bbdde55f4 P 2bcaf0a18b01483bb21c6edb93523299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:34.625551 23383 raft_consensus.cc:2258] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:34.625631 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22955
I20260925 07:24:34.626943 23383 ts_tablet_manager.cc:1905] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:34.628504 23383 ts_tablet_manager.cc:1918] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:34.628602 23383 log.cc:1196] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/d0c0d3d48f444c1092126b3bbdde55f4
I20260925 07:24:34.628963 23383 ts_tablet_manager.cc:1939] T d0c0d3d48f444c1092126b3bbdde55f4 P fea5cf672e2046fc8ca5553686639201: Deleting consensus metadata
I20260925 07:24:34.629412 22879 catalog_manager.cc:4841] TS fea5cf672e2046fc8ca5553686639201 (127.4.47.2:35373): tablet d0c0d3d48f444c1092126b3bbdde55f4 (table test-table [id=c1d055277618404eb6079a31a47e5a4b]) successfully deleted
I20260925 07:24:34.631213 23381 tablet_replica.cc:330] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: stopping tablet replica
I20260925 07:24:34.631393 23381 raft_consensus.cc:2229] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:34.631511 23381 raft_consensus.cc:2258] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:34.631922 23381 ts_tablet_manager.cc:1905] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:24:34.633190 22880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:37397 (error 108)
I20260925 07:24:34.633361 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23086
I20260925 07:24:34.634039 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:34.633610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:37397 (local address 127.0.0.1:58752)
0925 07:24:34.633961 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:34.633999 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
W20260925 07:24:34.634227 22880 catalog_manager.cc:4584] TS 2bcaf0a18b01483bb21c6edb93523299 (127.4.47.1:37397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0c0d3d48f444c1092126b3bbdde55f4: Network error: Client connection negotiation failed: client connection to 127.4.47.1:37397: connect: Connection refused (error 111)
I20260925 07:24:34.634620 23381 ts_tablet_manager.cc:1918] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:34.634716 23381 log.cc:1196] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/d0c0d3d48f444c1092126b3bbdde55f4
I20260925 07:24:34.634960 23381 ts_tablet_manager.cc:1939] T d0c0d3d48f444c1092126b3bbdde55f4 P d41a573d9371423987ba7be1de02bccc: Deleting consensus metadata
I20260925 07:24:34.635461 22882 catalog_manager.cc:4841] TS d41a573d9371423987ba7be1de02bccc (127.4.47.3:40953): tablet d0c0d3d48f444c1092126b3bbdde55f4 (table test-table [id=c1d055277618404eb6079a31a47e5a4b]) successfully deleted
I20260925 07:24:34.641433 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23217
I20260925 07:24:34.647609 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 22864
2026-09-25T07:24:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (913 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:34 dist-test-slave-zw07 krb5kdc[23387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:34 dist-test-slave-zw07 krb5kdc[23387](info): set up 2 sockets
Sep 25 07:24:34 dist-test-slave-zw07 krb5kdc[23387](info): commencing operation
krb5kdc: starting...
W20260925 07:24:35.690925 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:35 dist-test-slave-zw07 krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:35.818938 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36917
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:35.903348 23400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:35.903626 23400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:35.903666 23400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:35.904960 23400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:35.905047 23400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:35.905071 23400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:35.905094 23400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:35.905122 23400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:35.906689 23400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:35.906996 23400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:35.907276 23400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:35.912292 23405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:35.912315 23406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:35.912549 23400 server_base.cc:1034] running on GCE node
W20260925 07:24:35.912292 23408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:35.912880 23400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:35.913254 23400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:35.914425 23400 hybrid_clock.cc:648] HybridClock initialized: now 1790321075914420 us; error 56 us; skew 500 ppm
Sep 25 07:24:35 dist-test-slave-zw07 krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321075, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:35.917064 23400 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:35.917522 23400 webserver.cc:460] Webserver started at http://127.4.47.62:41331/ using document root <none> and password file <none>
I20260925 07:24:35.917718 23400 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:35.917760 23400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:35.917850 23400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:35.918877 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "aca72d61646c4a208f199b06d3eede5e"
format_stamp: "Formatted at 2026-09-25 07:24:35 on dist-test-slave-zw07"
server_key: "97e12d3633c18dc6ee05eba157fe5c84"
server_key_iv: "79d31249667c7fa52bfad476c7d4b5cc"
server_key_version: "encryptionkey@0"
I20260925 07:24:35.919260 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "aca72d61646c4a208f199b06d3eede5e"
format_stamp: "Formatted at 2026-09-25 07:24:35 on dist-test-slave-zw07"
server_key: "97e12d3633c18dc6ee05eba157fe5c84"
server_key_iv: "79d31249667c7fa52bfad476c7d4b5cc"
server_key_version: "encryptionkey@0"
I20260925 07:24:35.920568 23400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:35.921283 23415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:35.921485 23400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:35.921573 23400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "aca72d61646c4a208f199b06d3eede5e"
format_stamp: "Formatted at 2026-09-25 07:24:35 on dist-test-slave-zw07"
server_key: "97e12d3633c18dc6ee05eba157fe5c84"
server_key_iv: "79d31249667c7fa52bfad476c7d4b5cc"
server_key_version: "encryptionkey@0"
I20260925 07:24:35.921622 23400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:35.937256 23400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:35.939689 23400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:35.939807 23400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:35.943612 23400 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:36917
I20260925 07:24:35.943670 23467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:36917 every 8 connection(s)
I20260925 07:24:35.944023 23400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:35.944490 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23400
I20260925 07:24:35.944636 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:35.944846 4284 external_mini_cluster.cc:1428] Setting key bdcb071c19eba7ecc42fc18b7dd476ae
I20260925 07:24:35.945214 23468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:35.948779 23468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: Bootstrap starting.
I20260925 07:24:35.949501 23468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:35.949811 23468 log.cc:824] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:35.950562 23468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: No bootstrap required, opened a new log
Sep 25 07:24:35 dist-test-slave-zw07 krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:35.953050 23468 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } }
I20260925 07:24:35.953234 23468 raft_consensus.cc:374] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:35.953260 23468 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca72d61646c4a208f199b06d3eede5e, State: Initialized, Role: FOLLOWER
I20260925 07:24:35.953439 23468 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } }
I20260925 07:24:35.953537 23468 raft_consensus.cc:388] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:35.953574 23468 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:35.953634 23468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:35.954411 23468 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } }
I20260925 07:24:35.954524 23468 leader_election.cc:302] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aca72d61646c4a208f199b06d3eede5e; no voters:
I20260925 07:24:35.954713 23468 leader_election.cc:288] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:35.954800 23473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:35.954960 23473 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [term 1 LEADER]: Becoming Leader. State: Replica: aca72d61646c4a208f199b06d3eede5e, State: Running, Role: LEADER
I20260925 07:24:35.954989 23468 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:35.955050 23473 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } }
I20260925 07:24:35.955539 23474 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aca72d61646c4a208f199b06d3eede5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } } }
I20260925 07:24:35.955632 23474 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:35.955868 23475 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca72d61646c4a208f199b06d3eede5e. Latest consensus state: current_term: 1 leader_uuid: "aca72d61646c4a208f199b06d3eede5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca72d61646c4a208f199b06d3eede5e" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 36917 } } }
I20260925 07:24:35.956010 23475 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:35.956158 23482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:35.956717 23482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:35.957226 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:35.945383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.4.47.62:36917)
0925 07:24:35.945577 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:35.945586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:35.945714 (+ 128us) server_negotiation.cc:407] Connection header received
0925 07:24:35.946384 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:35.946392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:35.946442 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:35.946559 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:35.947401 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:35.948291 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:35.949351 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:35.949808 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:35.949930 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:35.952768 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:35.952773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:35.952773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:35.952788 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:35.955119 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:35.955577 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:35.955581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:35.955582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:35.955718 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:35.955954 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:35.955955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:35.955956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:35.956374 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:35.956589 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:35.956693 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:24:35.956987 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 07:24:35.958674 23482 catalog_manager.cc:1347] Generated new cluster ID: 7cdef993cbc9431087161bd685133412
I20260925 07:24:35.958725 23482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:35.965902 23482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:35.966478 23482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:35.972146 23482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: Generated new TSK 0
I20260925 07:24:35.972368 23482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.47.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
Entry for principal kudu/127.4.47.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab.
I20260925 07:24:36.001423 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:36.103896 23494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:36.104112 23494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:36.104137 23494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:36.105317 23494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:36.105360 23494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:36.105387 23494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:36.106817 23494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.1.keytab
--principal=kudu/127.4.47.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:36.107060 23494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:36.107344 23494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:36.112649 23499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.112879 23500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.112787 23502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.113260 23494 server_base.cc:1034] running on GCE node
I20260925 07:24:36.113482 23494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:36.113752 23494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:36.114923 23494 hybrid_clock.cc:648] HybridClock initialized: now 1790321076114908 us; error 28 us; skew 500 ppm
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:36.117586 23494 init.cc:377] Logged in from keytab as kudu/127.4.47.1@KRBTEST.COM (short username kudu)
I20260925 07:24:36.118127 23494 webserver.cc:460] Webserver started at http://127.4.47.1:42655/ using document root <none> and password file <none>
I20260925 07:24:36.118364 23494 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:36.118435 23494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:36.118593 23494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:36.119771 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "bf9e52425725478482843c958113de42"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c15c7a48aebde595eab1b197b9e2c694"
server_key_iv: "7aafef761bc056d1b26026333733e370"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.120157 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "bf9e52425725478482843c958113de42"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c15c7a48aebde595eab1b197b9e2c694"
server_key_iv: "7aafef761bc056d1b26026333733e370"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.121503 23494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:36.122236 23509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.122413 23494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:36.122520 23494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "bf9e52425725478482843c958113de42"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c15c7a48aebde595eab1b197b9e2c694"
server_key_iv: "7aafef761bc056d1b26026333733e370"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.122596 23494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:36.149551 23494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:36.151649 23494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:36.151846 23494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:36.152114 23494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:36.152459 23494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:36.152518 23494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.152563 23494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:36.152611 23494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.158941 23494 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:35141
I20260925 07:24:36.159020 23622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:35141 every 8 connection(s)
I20260925 07:24:36.159401 23494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.1@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:36.168093 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23494
I20260925 07:24:36.168232 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:36.168509 4284 external_mini_cluster.cc:1428] Setting key eb7650628497cfbfc09b9bbd93c8ecbe
I20260925 07:24:36.168989 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.160237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:57095 (local address 127.4.47.62:36917)
0925 07:24:36.160286 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.160288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.160498 (+ 210us) server_negotiation.cc:407] Connection header received
0925 07:24:36.160974 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.160975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.160985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.161049 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.162011 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.162511 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.163528 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.163726 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.163783 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.165833 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.165834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.165835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:36.165842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.167436 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.167788 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.167789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.167790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.167840 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.168073 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.168074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.168074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.168265 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.168380 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.168752 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.168876 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:24:36.169274 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.160197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36917 (local address 127.4.47.1:57095)
0925 07:24:36.160366 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.160404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.160884 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.161104 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.161108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.161713 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:36.161923 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.161928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.162595 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.162595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.163447 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.163449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.163809 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.163810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.163910 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.164405 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:36.164414 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:36.165745 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:36.167503 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.167509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.167511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.167721 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.167911 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.167912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.167912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.168008 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.168375 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:36.168386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:36.168533 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.168726 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.168953 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260925 07:24:36.169783 23623 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36917
I20260925 07:24:36.169906 23623 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:36.170085 23623 heartbeater.cc:511] Master 127.4.47.62:36917 requested a full tablet report, sending...
I20260925 07:24:36.170677 23432 ts_manager.cc:194] Registered new tserver with Master: bf9e52425725478482843c958113de42 (127.4.47.1:35141)
I20260925 07:24:36.171697 23432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.1@KRBTEST.COM'} at 127.4.47.1:57095
WARNING: no policy specified for kudu/127.4.47.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
Entry for principal kudu/127.4.47.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab.
I20260925 07:24:36.194260 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:36.277474 23628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:36.277704 23628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:36.277730 23628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:36.278949 23628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:36.278995 23628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:36.279023 23628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:36.280548 23628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.2.keytab
--principal=kudu/127.4.47.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:36.280817 23628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:36.281056 23628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:36.286336 23633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.286391 23636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.286602 23628 server_base.cc:1034] running on GCE node
W20260925 07:24:36.286352 23634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.286904 23628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:36.287177 23628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:36.288337 23628 hybrid_clock.cc:648] HybridClock initialized: now 1790321076288322 us; error 30 us; skew 500 ppm
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:36.290851 23628 init.cc:377] Logged in from keytab as kudu/127.4.47.2@KRBTEST.COM (short username kudu)
I20260925 07:24:36.291397 23628 webserver.cc:460] Webserver started at http://127.4.47.2:43845/ using document root <none> and password file <none>
I20260925 07:24:36.291604 23628 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:36.291647 23628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:36.291736 23628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:36.292769 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "1165d0941d43482ca96ee87f34a87924"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c170e0dadd684fa786afc4509aa31bde"
server_key_iv: "506457d27e84631e2d1f77ecd83b0482"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.293092 23628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "1165d0941d43482ca96ee87f34a87924"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c170e0dadd684fa786afc4509aa31bde"
server_key_iv: "506457d27e84631e2d1f77ecd83b0482"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.294353 23628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:36.295012 23643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.295257 23628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:36.295365 23628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "1165d0941d43482ca96ee87f34a87924"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "c170e0dadd684fa786afc4509aa31bde"
server_key_iv: "506457d27e84631e2d1f77ecd83b0482"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.295442 23628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:36.306267 23628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:36.308421 23628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:36.308595 23628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:36.308868 23628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:36.309233 23628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:36.309268 23628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.309291 23628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:36.309304 23628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.315310 23628 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:39623
I20260925 07:24:36.315393 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:39623 every 8 connection(s)
I20260925 07:24:36.315721 23628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:24:36.319489 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23628
I20260925 07:24:36.319617 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:36.319826 4284 external_mini_cluster.cc:1428] Setting key eb5acaf0f742658dac85ee7ab08931f4
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.2@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:36.326999 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.316516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:38723 (local address 127.4.47.62:36917)
0925 07:24:36.316577 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.316579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.316782 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:24:36.317251 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.317252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.317263 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.317317 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.318314 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.318904 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.320385 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.320710 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.320776 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.323649 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.323650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.323651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:36.323658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.325261 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.325917 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.325918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.325918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.325974 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.326293 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.326293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.326294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.326404 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.326455 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.326852 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.326929 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:24:36.327414 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.316466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36917 (local address 127.4.47.2:38723)
0925 07:24:36.316650 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.316688 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.317168 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.317382 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.317385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.317958 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:36.318218 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.318223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.318987 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.318988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.320280 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.320284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.320807 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.320809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.320979 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.321575 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:36.321586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:36.323547 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:36.325388 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.325398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.325401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.325835 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.326056 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.326057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.326057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.326226 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.326472 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:36.326480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:36.326685 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.326826 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.327034 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260925 07:24:36.328018 23757 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36917
I20260925 07:24:36.328159 23757 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:36.328414 23757 heartbeater.cc:511] Master 127.4.47.62:36917 requested a full tablet report, sending...
I20260925 07:24:36.328840 23432 ts_manager.cc:194] Registered new tserver with Master: 1165d0941d43482ca96ee87f34a87924 (127.4.47.2:39623)
I20260925 07:24:36.329306 23432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.2@KRBTEST.COM'} at 127.4.47.2:38723
WARNING: no policy specified for kudu/127.4.47.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
Entry for principal kudu/127.4.47.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab.
I20260925 07:24:36.350052 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:36.435403 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:36.435645 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:36.435673 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:36.436870 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:36.436911 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:36.436939 23762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:36.438397 23762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.3.keytab
--principal=kudu/127.4.47.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:36.438596 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:36.438786 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:36.444090 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.444132 23762 server_base.cc:1034] running on GCE node
W20260925 07:24:36.444058 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.444063 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.444551 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:36.444783 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:36.445933 23762 hybrid_clock.cc:648] HybridClock initialized: now 1790321076445915 us; error 28 us; skew 500 ppm
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:36.448733 23762 init.cc:377] Logged in from keytab as kudu/127.4.47.3@KRBTEST.COM (short username kudu)
I20260925 07:24:36.449201 23762 webserver.cc:460] Webserver started at http://127.4.47.3:46145/ using document root <none> and password file <none>
I20260925 07:24:36.449438 23762 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:36.449506 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:36.449669 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:36.450807 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "55e1b71e5b040d4158a91d35e18f3ee1"
server_key_iv: "d4bcb28cd1896e1cb7a878a90427042b"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.451236 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "55e1b71e5b040d4158a91d35e18f3ee1"
server_key_iv: "d4bcb28cd1896e1cb7a878a90427042b"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.452663 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:36.453472 23777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.453656 23762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:36.453760 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "55e1b71e5b040d4158a91d35e18f3ee1"
server_key_iv: "d4bcb28cd1896e1cb7a878a90427042b"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.453850 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:36.479928 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:36.482424 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:36.482607 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:36.482863 23762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:36.483264 23762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:36.483319 23762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.483355 23762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:36.483383 23762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.489305 23762 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:37385
I20260925 07:24:36.489348 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:37385 every 8 connection(s)
I20260925 07:24:36.489681 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:36.495798 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23762
I20260925 07:24:36.495936 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:36.496125 4284 external_mini_cluster.cc:1428] Setting key 7fcb9d34712e276b7283371fcba514cb
Sep 25 07:24:36 dist-test-slave-zw07 krb5kdc[23387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321076, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.3@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:36.499729 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.490389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:55941 (local address 127.4.47.62:36917)
0925 07:24:36.490436 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.490438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.490697 (+ 259us) server_negotiation.cc:407] Connection header received
0925 07:24:36.491322 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.491323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.491337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.491390 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.492394 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.492909 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.493950 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.494159 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.494248 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.496832 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.496833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.496834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:36.496842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.498379 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.498748 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.498748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.498749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.498799 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.499019 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.499020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.499020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.499215 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.499299 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.499599 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.499663 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:24:36.500011 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.490381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:36917 (local address 127.4.47.3:55941)
0925 07:24:36.490560 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.490603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.491211 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.491476 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.491480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.492079 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:36.492280 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.492284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.493011 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.493013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.493851 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.493853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.494258 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.494259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.494397 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.494902 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:24:36.494910 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:36.496748 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:36.498450 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.498456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.498458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.498680 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.498863 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:24:36.498863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:24:36.498863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:24:36.498963 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:24:36.499333 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:36.499339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:36.499480 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.499571 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.499697 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260925 07:24:36.500684 23891 heartbeater.cc:347] Connected to a master server at 127.4.47.62:36917
I20260925 07:24:36.500770 23891 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:36.500967 23891 heartbeater.cc:511] Master 127.4.47.62:36917 requested a full tablet report, sending...
I20260925 07:24:36.501423 23432 ts_manager.cc:194] Registered new tserver with Master: 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e (127.4.47.3:37385)
I20260925 07:24:36.501849 23432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.47.3@KRBTEST.COM'} at 127.4.47.3:55941
I20260925 07:24:36.508255 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:36.515945 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.509743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.4.47.62:36917)
0925 07:24:36.509797 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.509799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.509913 (+ 114us) server_negotiation.cc:407] Connection header received
0925 07:24:36.509957 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.509958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.509967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.510020 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.510570 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.511180 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.511917 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.512131 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.512259 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.513461 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.513462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.513462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:36.513468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.514918 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.515297 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.515299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.515299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.515354 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:36.515507 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:36.515507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:36.515507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:36.515649 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.515721 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.515808 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.515877 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:24:36.517719 23432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:36.519606 23432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:36.526844 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.4.47.1:35141)
0925 07:24:36.522513 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.522517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.522528 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:24:36.522742 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.522745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.522763 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.522848 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:36.523195 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.523983 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.525665 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526339 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526507 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.526707 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.526747 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":84,"threads_started":1}
I20260925 07:24:36.526844 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:35141 (local address 127.0.0.1:49832)
0925 07:24:36.522476 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.522484 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.522543 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.522948 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.522951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.522956 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:36.523113 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.523114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.524105 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.524106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.525569 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.525572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526442 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.526587 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526600 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.526688 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.526747 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":74,"thread_start_us":51,"threads_started":1}
I20260925 07:24:36.527036 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.4.47.2:39623)
0925 07:24:36.522787 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.522792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.522801 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:24:36.522840 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.522843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.522859 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.522936 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:36.523242 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.524323 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.525690 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526454 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526525 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526587 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.526887 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.526930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260925 07:24:36.527252 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:37385 (local address 127.0.0.1:58208)
0925 07:24:36.522396 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.522444 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.522678 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.522948 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.522951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.522954 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:36.523108 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.523110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.524499 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.524500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.525680 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.525682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526620 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.526696 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526704 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.526757 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.526816 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":92,"thread_start_us":156,"threads_started":1}
I20260925 07:24:36.527411 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.4.47.3:37385)
0925 07:24:36.522409 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.522415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.522520 (+ 105us) server_negotiation.cc:407] Connection header received
0925 07:24:36.522772 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.522774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.522792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.522843 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:36.523183 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.524371 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.525755 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526454 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526525 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526579 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.527196 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.527241 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260925 07:24:36.527438 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.522341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:39623 (local address 127.0.0.1:56384)
0925 07:24:36.522565 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.522573 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.522610 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.523015 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.523016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.523018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:36.523133 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.523134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.524530 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.524531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.525557 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.525589 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.526618 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.526619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.526696 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.526705 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.526757 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.526816 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":11,"thread_start_us":77,"threads_started":1}
I20260925 07:24:36.527930 23557 tablet_service.cc:1467] Processing CreateTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d (DEFAULT_TABLE table=test-table [id=f960721751054a9987300f7f384bf136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:36.528054 23691 tablet_service.cc:1467] Processing CreateTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d (DEFAULT_TABLE table=test-table [id=f960721751054a9987300f7f384bf136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:36.528118 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d (DEFAULT_TABLE table=test-table [id=f960721751054a9987300f7f384bf136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:36.528364 23557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8923c08a279349f28f8f2cc7bdb9625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:36.528419 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8923c08a279349f28f8f2cc7bdb9625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:36.528437 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8923c08a279349f28f8f2cc7bdb9625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:36.530989 23908 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Bootstrap starting.
I20260925 07:24:36.531749 23908 tablet_bootstrap.cc:654] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:36.531746 23909 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Bootstrap starting.
I20260925 07:24:36.532114 23910 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: Bootstrap starting.
I20260925 07:24:36.532455 23909 tablet_bootstrap.cc:654] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:36.532472 23908 log.cc:824] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:36.532759 23910 tablet_bootstrap.cc:654] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:36.533080 23909 log.cc:824] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:36.533329 23910 log.cc:824] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:36.533373 23908 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: No bootstrap required, opened a new log
I20260925 07:24:36.533454 23908 ts_tablet_manager.cc:1397] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:24:36.533841 23909 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: No bootstrap required, opened a new log
I20260925 07:24:36.533912 23909 ts_tablet_manager.cc:1397] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:36.534034 23910 tablet_bootstrap.cc:492] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: No bootstrap required, opened a new log
I20260925 07:24:36.534127 23910 ts_tablet_manager.cc:1397] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:36.535511 23908 raft_consensus.cc:348] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.535704 23908 raft_consensus.cc:374] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:36.535741 23908 raft_consensus.cc:729] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1165d0941d43482ca96ee87f34a87924, State: Initialized, Role: FOLLOWER
I20260925 07:24:36.535866 23908 consensus_queue.cc:260] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.535864 23909 raft_consensus.cc:348] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.536021 23909 raft_consensus.cc:374] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:36.535823 23910 raft_consensus.cc:348] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.536059 23909 raft_consensus.cc:729] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e, State: Initialized, Role: FOLLOWER
I20260925 07:24:36.536077 23910 raft_consensus.cc:374] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:36.536123 23910 raft_consensus.cc:729] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9e52425725478482843c958113de42, State: Initialized, Role: FOLLOWER
I20260925 07:24:36.536147 23908 ts_tablet_manager.cc:1428] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:36.536270 23757 heartbeater.cc:503] Master 127.4.47.62:36917 was elected leader, sending a full tablet report...
I20260925 07:24:36.536247 23910 consensus_queue.cc:260] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.536181 23909 consensus_queue.cc:260] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.536659 23623 heartbeater.cc:503] Master 127.4.47.62:36917 was elected leader, sending a full tablet report...
I20260925 07:24:36.536674 23910 ts_tablet_manager.cc:1428] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:36.536805 23909 ts_tablet_manager.cc:1428] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:36.536945 23891 heartbeater.cc:503] Master 127.4.47.62:36917 was elected leader, sending a full tablet report...
I20260925 07:24:36.553233 23914 raft_consensus.cc:482] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:36.553360 23914 raft_consensus.cc:504] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.553814 23914 leader_election.cc:288] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9e52425725478482843c958113de42 (127.4.47.1:35141), 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e (127.4.47.3:37385)
I20260925 07:24:36.557950 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.554073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:55947 (local address 127.4.47.3:37385)
0925 07:24:36.554134 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.554137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.554149 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:24:36.554180 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.554181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.554191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.554273 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:36.554438 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.555225 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.556942 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.557576 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.557649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.557660 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.557836 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.557877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:24:36.557950 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.554068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:42563 (local address 127.4.47.1:35141)
0925 07:24:36.554152 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.554154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.554285 (+ 131us) server_negotiation.cc:407] Connection header received
0925 07:24:36.554319 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.554320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.554331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.554379 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:36.554494 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.555275 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.556942 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.557560 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.557649 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.557660 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.557836 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.557877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 07:24:36.557986 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.553982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:37385 (local address 127.4.47.2:55947)
0925 07:24:36.554049 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.554062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.554110 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.554278 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.554279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.554281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:36.554363 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.554365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.555302 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.555303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.556752 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.556754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.557661 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.557662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.557745 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.557749 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.557814 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.557864 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":65}
I20260925 07:24:36.558198 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.553998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:35141 (local address 127.4.47.2:42563)
0925 07:24:36.554203 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.554210 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.554275 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.554381 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.554382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.554383 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:36.554444 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.554445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.555344 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.555344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.556752 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.556754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.557661 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.557662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.557745 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.557749 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.557830 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.557864 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":75,"thread_start_us":118,"threads_started":1}
I20260925 07:24:36.558354 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8923c08a279349f28f8f2cc7bdb9625d" candidate_uuid: "1165d0941d43482ca96ee87f34a87924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" is_pre_election: true
I20260925 07:24:36.558525 23845 raft_consensus.cc:2454] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1165d0941d43482ca96ee87f34a87924 in term 0.
I20260925 07:24:36.558604 23577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8923c08a279349f28f8f2cc7bdb9625d" candidate_uuid: "1165d0941d43482ca96ee87f34a87924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9e52425725478482843c958113de42" is_pre_election: true
I20260925 07:24:36.558755 23577 raft_consensus.cc:2454] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1165d0941d43482ca96ee87f34a87924 in term 0.
I20260925 07:24:36.558786 23646 leader_election.cc:302] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1165d0941d43482ca96ee87f34a87924, 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e; no voters:
I20260925 07:24:36.558957 23914 raft_consensus.cc:2781] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:36.559012 23914 raft_consensus.cc:482] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:36.559046 23914 raft_consensus.cc:3037] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:36.559795 23914 raft_consensus.cc:504] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.559949 23914 leader_election.cc:288] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [CANDIDATE]: Term 1 election: Requested vote from peers bf9e52425725478482843c958113de42 (127.4.47.1:35141), 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e (127.4.47.3:37385)
I20260925 07:24:36.560104 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8923c08a279349f28f8f2cc7bdb9625d" candidate_uuid: "1165d0941d43482ca96ee87f34a87924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e"
I20260925 07:24:36.560115 23577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8923c08a279349f28f8f2cc7bdb9625d" candidate_uuid: "1165d0941d43482ca96ee87f34a87924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9e52425725478482843c958113de42"
I20260925 07:24:36.560190 23845 raft_consensus.cc:3037] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:36.560213 23577 raft_consensus.cc:3037] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:36.560904 23577 raft_consensus.cc:2454] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1165d0941d43482ca96ee87f34a87924 in term 1.
I20260925 07:24:36.560971 23845 raft_consensus.cc:2454] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1165d0941d43482ca96ee87f34a87924 in term 1.
I20260925 07:24:36.561131 23644 leader_election.cc:302] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1165d0941d43482ca96ee87f34a87924, bf9e52425725478482843c958113de42; no voters:
I20260925 07:24:36.561241 23914 raft_consensus.cc:2781] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:36.561437 23914 raft_consensus.cc:686] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 1 LEADER]: Becoming Leader. State: Replica: 1165d0941d43482ca96ee87f34a87924, State: Running, Role: LEADER
I20260925 07:24:36.561568 23914 consensus_queue.cc:237] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } }
I20260925 07:24:36.562546 23432 catalog_manager.cc:5495] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1165d0941d43482ca96ee87f34a87924 (127.4.47.2). New cstate: current_term: 1 leader_uuid: "1165d0941d43482ca96ee87f34a87924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1165d0941d43482ca96ee87f34a87924" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 39623 } health_report { overall_health: HEALTHY } } }
W20260925 07:24:36.566291 23758 tablet.cc:2369] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:36.572952 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.570659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.4.47.2:39623)
0925 07:24:36.570820 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.570822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.570830 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:36.570859 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.570860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.570878 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.570938 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:36.571051 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.571607 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.572325 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.572505 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.572560 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.572682 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:36.572777 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:36.572838 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.572880 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.572896 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 07:24:36.574720 23845 raft_consensus.cc:1261] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 1 FOLLOWER]: Refusing update from remote peer 1165d0941d43482ca96ee87f34a87924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:36.574739 23577 raft_consensus.cc:1261] T 8923c08a279349f28f8f2cc7bdb9625d P bf9e52425725478482843c958113de42 [term 1 FOLLOWER]: Refusing update from remote peer 1165d0941d43482ca96ee87f34a87924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:36.575156 23918 consensus_queue.cc:1035] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9e52425725478482843c958113de42" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 35141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:36.575295 23914 consensus_queue.cc:1035] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fdfc8b9ecec43cfbbeeeffa2b1d6f5e" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:36.578367 23921 mvcc.cc:205] Tried to move back new op lower bound from 7333155129647296512 to 7333155129596948480. Current Snapshot: MvccSnapshot[applied={T|T < 7333155129647296512}]
I20260925 07:24:36.578496 23922 mvcc.cc:205] Tried to move back new op lower bound from 7333155129647296512 to 7333155129596948480. Current Snapshot: MvccSnapshot[applied={T|T < 7333155129647296512 or (T in {7333155129647296512})}]
I20260925 07:24:36.580703 23431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:36.580816 23431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:36.581642 23431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aca72d61646c4a208f199b06d3eede5e: Sending DeleteTablet for 3 replicas of tablet 8923c08a279349f28f8f2cc7bdb9625d
I20260925 07:24:36.581977 23557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:36 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:49832
I20260925 07:24:36.582064 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:36 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:58208
I20260925 07:24:36.582194 23691 tablet_service.cc:1514] Processing DeleteTablet for tablet 8923c08a279349f28f8f2cc7bdb9625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:36 UTC) from {username='kudu', principal='kudu/127.4.47.62@KRBTEST.COM'} at 127.0.0.1:56384
I20260925 07:24:36.582203 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23494
I20260925 07:24:36.584026 23933 tablet_replica.cc:330] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: stopping tablet replica
I20260925 07:24:36.584126 23933 raft_consensus.cc:2229] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:36.584249 23933 raft_consensus.cc:2258] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:36.584719 23933 ts_tablet_manager.cc:1905] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:36.586892 23933 ts_tablet_manager.cc:1918] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:36.586974 23933 log.cc:1196] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-1/wal/wals/8923c08a279349f28f8f2cc7bdb9625d
I20260925 07:24:36.587293 23933 ts_tablet_manager.cc:1939] T 8923c08a279349f28f8f2cc7bdb9625d P 1165d0941d43482ca96ee87f34a87924: Deleting consensus metadata
I20260925 07:24:36.587941 23418 catalog_manager.cc:4841] TS 1165d0941d43482ca96ee87f34a87924 (127.4.47.2:39623): tablet 8923c08a279349f28f8f2cc7bdb9625d (table test-table [id=f960721751054a9987300f7f384bf136]) successfully deleted
W20260925 07:24:36.588393 23416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:35141 (error 108)
I20260925 07:24:36.588899 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.588767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:35141 (local address 127.0.0.1:49842)
0925 07:24:36.588822 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.588859 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:24:36.589003 23416 catalog_manager.cc:4584] TS bf9e52425725478482843c958113de42 (127.4.47.1:35141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8923c08a279349f28f8f2cc7bdb9625d: Network error: Client connection negotiation failed: client connection to 127.4.47.1:35141: connect: Connection refused (error 111)
I20260925 07:24:36.589325 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23628
I20260925 07:24:36.589980 23932 tablet_replica.cc:330] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: stopping tablet replica
I20260925 07:24:36.590114 23932 raft_consensus.cc:2229] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:36.590240 23932 raft_consensus.cc:2258] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:36.590632 23932 ts_tablet_manager.cc:1905] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:36.592669 23932 ts_tablet_manager.cc:1918] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:36.592782 23932 log.cc:1196] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/8923c08a279349f28f8f2cc7bdb9625d
I20260925 07:24:36.593036 23932 ts_tablet_manager.cc:1939] T 8923c08a279349f28f8f2cc7bdb9625d P 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e: Deleting consensus metadata
I20260925 07:24:36.593585 23418 catalog_manager.cc:4841] TS 6fdfc8b9ecec43cfbbeeeffa2b1d6f5e (127.4.47.3:37385): tablet 8923c08a279349f28f8f2cc7bdb9625d (table test-table [id=f960721751054a9987300f7f384bf136]) successfully deleted
I20260925 07:24:36.595523 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23762
I20260925 07:24:36.601007 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23400
2026-09-25T07:24:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1954 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T07:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:36Z Disabled control of system clock
I20260925 07:24:36.629266 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:42241
--webserver_interface=127.4.47.62
--webserver_port=0
--builtin_ntp_servers=127.4.47.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:36.714594 23938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:36.714833 23938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:36.714856 23938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:36.716150 23938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:36.716197 23938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:36.716209 23938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:36.716223 23938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:36.716275 23938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:36.717754 23938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:42241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:36.718024 23938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:36.718334 23938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:36.723263 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.723269 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.723398 23938 server_base.cc:1034] running on GCE node
W20260925 07:24:36.723323 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.723770 23938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:36.724021 23938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:36.725159 23938 hybrid_clock.cc:648] HybridClock initialized: now 1790321076725146 us; error 29 us; skew 500 ppm
I20260925 07:24:36.726369 23938 webserver.cc:460] Webserver started at http://127.4.47.62:36255/ using document root <none> and password file <none>
I20260925 07:24:36.726567 23938 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:36.726608 23938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:36.726697 23938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:36.727811 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "fab014d319a54ad8a1886b8fbebedd68"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "1693ed34fb9432773848fe4596d864e5"
server_key_iv: "f7c916de3214b4da3f8553712d5ca275"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.728111 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "fab014d319a54ad8a1886b8fbebedd68"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "1693ed34fb9432773848fe4596d864e5"
server_key_iv: "f7c916de3214b4da3f8553712d5ca275"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.729354 23938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:36.730064 23952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.730242 23938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:36.730347 23938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "fab014d319a54ad8a1886b8fbebedd68"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "1693ed34fb9432773848fe4596d864e5"
server_key_iv: "f7c916de3214b4da3f8553712d5ca275"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.730423 23938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:36.743166 23938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:36.743500 23938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:36.743686 23938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:36.747939 23938 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:42241
I20260925 07:24:36.747979 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:42241 every 8 connection(s)
I20260925 07:24:36.748351 23938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:36.749328 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:36.751694 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: Bootstrap starting.
I20260925 07:24:36.752252 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:36.752516 24005 log.cc:824] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:36.753146 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: No bootstrap required, opened a new log
I20260925 07:24:36.753899 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 23938
I20260925 07:24:36.754024 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:36.754202 4284 external_mini_cluster.cc:1428] Setting key 3cb9c71ed1be185d1262d46fbcf24ecf
I20260925 07:24:36.755349 24005 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } }
I20260925 07:24:36.755549 24005 raft_consensus.cc:374] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:36.755590 24005 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab014d319a54ad8a1886b8fbebedd68, State: Initialized, Role: FOLLOWER
I20260925 07:24:36.755743 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } }
I20260925 07:24:36.755841 24005 raft_consensus.cc:388] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:36.755872 24005 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:36.755923 24005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:36.756665 24005 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } }
I20260925 07:24:36.756795 24005 leader_election.cc:302] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fab014d319a54ad8a1886b8fbebedd68; no voters:
I20260925 07:24:36.756960 24005 leader_election.cc:288] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:36.757048 24010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:36.757194 24010 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [term 1 LEADER]: Becoming Leader. State: Replica: fab014d319a54ad8a1886b8fbebedd68, State: Running, Role: LEADER
I20260925 07:24:36.757272 24005 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:36.757324 24010 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } }
I20260925 07:24:36.757644 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fab014d319a54ad8a1886b8fbebedd68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } } }
I20260925 07:24:36.757732 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:36.757959 24016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:36.757979 24012 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab014d319a54ad8a1886b8fbebedd68. Latest consensus state: current_term: 1 leader_uuid: "fab014d319a54ad8a1886b8fbebedd68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab014d319a54ad8a1886b8fbebedd68" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 42241 } } }
I20260925 07:24:36.758060 24012 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:36.758714 24016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:36.760536 24016 catalog_manager.cc:1347] Generated new cluster ID: b062d5a500534f829b3a486954dbf996
I20260925 07:24:36.760620 24016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:36.761065 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.754768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.4.47.62:42241)
0925 07:24:36.755009 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.755019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.755055 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:24:36.755296 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.755304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.755390 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.755542 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.755823 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.757621 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.758713 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.759398 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.759516 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.760238 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.760241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.760242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:36.760448 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.760489 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:36.760505 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.760567 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.760763 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.760850 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260925 07:24:36.777853 24016 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:36.778659 24016 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:36.792089 24016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: Generated new TSK 0
I20260925 07:24:36.792325 24016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:36.804781 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.1:0
--local_ip_for_outbound_sockets=127.4.47.1
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--builtin_ntp_servers=127.4.47.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:24:36.887912 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:36.888139 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:36.888164 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:36.889509 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:36.889551 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:36.889585 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.1
I20260925 07:24:36.890992 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.47.1
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.1
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:36.891269 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:36.891551 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:36.896600 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.896662 24029 server_base.cc:1034] running on GCE node
W20260925 07:24:36.896605 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:36.896600 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:36.897070 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:36.897286 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:36.898430 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790321076898416 us; error 28 us; skew 500 ppm
I20260925 07:24:36.899873 24029 webserver.cc:460] Webserver started at http://127.4.47.1:32927/ using document root <none> and password file <none>
I20260925 07:24:36.900156 24029 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:36.900207 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:36.900378 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:36.901458 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data/instance:
uuid: "146836e64c104cdd9bfa77c519299d5c"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "156da727de49bf3518923edb8dd24ede"
server_key_iv: "4f620a4f28b5f554e1a1091c6e126a16"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.901822 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance:
uuid: "146836e64c104cdd9bfa77c519299d5c"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "156da727de49bf3518923edb8dd24ede"
server_key_iv: "4f620a4f28b5f554e1a1091c6e126a16"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.903264 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:36.904014 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.904224 24029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:36.904325 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal
uuid: "146836e64c104cdd9bfa77c519299d5c"
format_stamp: "Formatted at 2026-09-25 07:24:36 on dist-test-slave-zw07"
server_key: "156da727de49bf3518923edb8dd24ede"
server_key_iv: "4f620a4f28b5f554e1a1091c6e126a16"
server_key_version: "encryptionkey@0"
I20260925 07:24:36.904410 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:36.936661 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:36.937011 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:36.937177 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:36.937423 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:36.937773 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:36.937840 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.937883 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:36.937911 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:36.944278 24029 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.1:38133
I20260925 07:24:36.944324 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.1:38133 every 8 connection(s)
I20260925 07:24:36.944715 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/data/info.pb
I20260925 07:24:36.950443 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24029
I20260925 07:24:36.950557 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-0/wal/instance
I20260925 07:24:36.950801 4284 external_mini_cluster.cc:1428] Setting key 3f478d0df463951f32b814f1a7f864f4
I20260925 07:24:36.950907 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.945688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.1:50225 (local address 127.4.47.62:42241)
0925 07:24:36.945737 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:36.945739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:36.946019 (+ 280us) server_negotiation.cc:407] Connection header received
0925 07:24:36.946650 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:36.946652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:36.946666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:36.946717 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:36.947519 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.948060 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.949110 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.949341 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.949433 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.950449 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:36.950450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:36.950451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:36.950460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:36.950479 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:36.950483 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:36.950596 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:36.950765 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 07:24:36.950799 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:24:36.951175 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:36.945685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42241 (local address 127.4.47.1:50225)
0925 07:24:36.945892 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:36.945930 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:36.946513 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:36.946801 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:36.946805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:36.946813 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:36.947419 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.947426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.948175 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.948176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.949000 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:36.949002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:36.949448 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:36.949449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:36.949592 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:36.950240 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:36.950241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:36.950287 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:36.950323 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:36.950607 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:36.950614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:36.950618 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:36.950746 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 07:24:36.950880 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260925 07:24:36.951654 24157 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42241
I20260925 07:24:36.951794 24157 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:36.951992 24157 heartbeater.cc:511] Master 127.4.47.62:42241 requested a full tablet report, sending...
I20260925 07:24:36.951998 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.2:0
--local_ip_for_outbound_sockets=127.4.47.2
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--builtin_ntp_servers=127.4.47.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:24:36.952611 23968 ts_manager.cc:194] Registered new tserver with Master: 146836e64c104cdd9bfa77c519299d5c (127.4.47.1:38133)
I20260925 07:24:36.953560 23968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.1:50225
W20260925 07:24:37.036020 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:37.036258 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:37.036281 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:37.037488 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:37.037529 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:37.037556 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.2
I20260925 07:24:37.038887 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.47.2
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.2
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:37.039187 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:37.039459 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:37.044611 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:37.044606 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:37.044606 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:37.044869 24160 server_base.cc:1034] running on GCE node
I20260925 07:24:37.045181 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:37.045424 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:37.046619 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790321077046601 us; error 27 us; skew 500 ppm
I20260925 07:24:37.048118 24160 webserver.cc:460] Webserver started at http://127.4.47.2:40719/ using document root <none> and password file <none>
I20260925 07:24:37.048396 24160 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:37.048472 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:37.048638 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:37.049779 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data/instance:
uuid: "5b25cfea905249cdbf1221a8efbb79fe"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "4aaa4b024f0e5a6fad847f1d769cebf4"
server_key_iv: "f8944c0f0f74af35405e79e547410e33"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.050170 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance:
uuid: "5b25cfea905249cdbf1221a8efbb79fe"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "4aaa4b024f0e5a6fad847f1d769cebf4"
server_key_iv: "f8944c0f0f74af35405e79e547410e33"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.051753 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:37.052591 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.052770 24160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:37.052879 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal
uuid: "5b25cfea905249cdbf1221a8efbb79fe"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "4aaa4b024f0e5a6fad847f1d769cebf4"
server_key_iv: "f8944c0f0f74af35405e79e547410e33"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.052986 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:37.069357 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:37.069725 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:37.069960 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:37.070230 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:37.070574 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:37.070638 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.070710 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:37.070740 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.076748 24160 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.2:43039
I20260925 07:24:37.076794 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.2:43039 every 8 connection(s)
I20260925 07:24:37.077131 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/data/info.pb
I20260925 07:24:37.077730 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24160
I20260925 07:24:37.077863 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-1/wal/instance
I20260925 07:24:37.078089 4284 external_mini_cluster.cc:1428] Setting key 608061286524704587ae55375cb6c1de
I20260925 07:24:37.079509 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.47.3:0
--local_ip_for_outbound_sockets=127.4.47.3
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--builtin_ntp_servers=127.4.47.20:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:24:37.083498 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.077907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:52823 (local address 127.4.47.62:42241)
0925 07:24:37.077965 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.077967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.078295 (+ 328us) server_negotiation.cc:407] Connection header received
0925 07:24:37.078984 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.078985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.079002 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.079101 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:37.080091 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.080599 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.081754 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.082071 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.082156 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.083110 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:37.083111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:37.083112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:37.083117 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:37.083144 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:37.083150 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:37.083214 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.083328 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.083368 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:24:37.083612 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.077898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42241 (local address 127.4.47.2:52823)
0925 07:24:37.078134 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.078189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.078879 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.079191 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.079196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.079207 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:37.079972 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.079982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.080702 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.080704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.081652 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.081654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.082173 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.082174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.082302 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.082933 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:37.082933 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:37.082964 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:37.082989 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:37.083229 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:37.083236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:37.083240 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.083319 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.083407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260925 07:24:37.084233 24288 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42241
I20260925 07:24:37.084376 24288 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:37.084586 24288 heartbeater.cc:511] Master 127.4.47.62:42241 requested a full tablet report, sending...
I20260925 07:24:37.085011 23968 ts_manager.cc:194] Registered new tserver with Master: 5b25cfea905249cdbf1221a8efbb79fe (127.4.47.2:43039)
I20260925 07:24:37.085582 23968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.2:52823
W20260925 07:24:37.163467 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:37.163691 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:37.163714 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:37.164844 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:37.164883 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:37.164909 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.47.3
I20260925 07:24:37.166209 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.47.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.47.3
--webserver_port=0
--tserver_master_addrs=127.4.47.62:42241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.47.3
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:37.166406 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:37.166677 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:37.171799 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:37.171770 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:37.171996 24291 server_base.cc:1034] running on GCE node
W20260925 07:24:37.172048 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:37.172322 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:37.172613 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:37.173764 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790321077173735 us; error 41 us; skew 500 ppm
I20260925 07:24:37.175220 24291 webserver.cc:460] Webserver started at http://127.4.47.3:40149/ using document root <none> and password file <none>
I20260925 07:24:37.175489 24291 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:37.175539 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:37.175678 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:37.176784 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data/instance:
uuid: "ebf6c1c5cc034cd8baace0de1088f5c8"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "d128cee7ec42527dba59ab96a71072de"
server_key_iv: "21de83fcf14c4baca9b7e463f970d891"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.177173 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance:
uuid: "ebf6c1c5cc034cd8baace0de1088f5c8"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "d128cee7ec42527dba59ab96a71072de"
server_key_iv: "21de83fcf14c4baca9b7e463f970d891"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.178524 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:37.179458 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.179615 24291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.179695 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal
uuid: "ebf6c1c5cc034cd8baace0de1088f5c8"
format_stamp: "Formatted at 2026-09-25 07:24:37 on dist-test-slave-zw07"
server_key: "d128cee7ec42527dba59ab96a71072de"
server_key_iv: "21de83fcf14c4baca9b7e463f970d891"
server_key_version: "encryptionkey@0"
I20260925 07:24:37.179762 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:37.188400 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:37.188674 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:37.188791 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:37.188985 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:24:37.189337 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:24:37.189399 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.189441 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:24:37.189469 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:37.195598 24291 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.3:43301
I20260925 07:24:37.195742 24418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.3:43301 every 8 connection(s)
I20260925 07:24:37.196048 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/data/info.pb
I20260925 07:24:37.202152 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.197185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.3:44141 (local address 127.4.47.62:42241)
0925 07:24:37.197236 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.197238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.197895 (+ 657us) server_negotiation.cc:407] Connection header received
0925 07:24:37.198400 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.198401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.198412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.198467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:37.199217 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.199749 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.200773 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.200978 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.201040 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.201894 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:37.201895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:37.201896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:37.201901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:37.201916 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:37.201920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:37.201970 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.202050 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.202080 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:24:37.202397 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.196840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:42241 (local address 127.4.47.3:44141)
0925 07:24:37.197743 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.197792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.198305 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.198523 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.198526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.198534 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:24:37.199049 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.199053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.199836 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.199837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.200681 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.200683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.201061 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.201062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.201173 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.201750 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:24:37.201750 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:24:37.201780 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:24:37.201806 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:24:37.201980 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:24:37.201985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:24:37.201987 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.202040 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.202122 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":82,"threads_started":1}
I20260925 07:24:37.202912 24419 heartbeater.cc:347] Connected to a master server at 127.4.47.62:42241
I20260925 07:24:37.203019 24419 heartbeater.cc:464] Registering TS with master...
I20260925 07:24:37.203274 24419 heartbeater.cc:511] Master 127.4.47.62:42241 requested a full tablet report, sending...
I20260925 07:24:37.203779 23968 ts_manager.cc:194] Registered new tserver with Master: ebf6c1c5cc034cd8baace0de1088f5c8 (127.4.47.3:43301)
I20260925 07:24:37.204133 23968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.47.3:44141
I20260925 07:24:37.204993 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24291
I20260925 07:24:37.205085 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal/instance
I20260925 07:24:37.205228 4284 external_mini_cluster.cc:1428] Setting key fb02e4cdc6687857907381bc8d3a58f4
I20260925 07:24:37.205940 4284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:24:37.210175 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.207453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.4.47.62:42241)
0925 07:24:37.207500 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.207501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.207658 (+ 157us) server_negotiation.cc:407] Connection header received
0925 07:24:37.207690 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.207690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.207707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.207756 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:37.207905 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.208369 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.209085 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.209309 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.209365 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.209952 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:37.209954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:37.209954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:37.209957 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:37.209970 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:37.209973 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:37.210014 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.210061 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.210081 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:24:37.212005 23968 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:24:37.213541 23968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:24:37.219995 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.215856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43301 (local address 127.0.0.1:34698)
0925 07:24:37.216226 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.216238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.216282 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.216495 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.216498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.216500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:37.216579 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.216581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.217593 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.217594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.218850 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.218852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.219685 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.219686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.219775 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.219788 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.219865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.219916 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":114,"threads_started":1}
I20260925 07:24:37.220008 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.216114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.4.47.3:43301)
0925 07:24:37.216292 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.216297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.216317 (+ 20us) server_negotiation.cc:407] Connection header received
0925 07:24:37.216361 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.216363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.216380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.216454 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:37.216649 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.217507 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.218948 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.219589 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.219686 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.219788 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.219881 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.219919 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260925 07:24:37.220343 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.215856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:43039 (local address 127.0.0.1:46890)
0925 07:24:37.216331 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.216342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.216401 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.216893 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.216894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.216897 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:37.216936 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.216937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.218112 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.218113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.219153 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.219156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.220090 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.220092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.220176 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.220182 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.220216 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.220255 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260925 07:24:37.220645 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.216032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.4.47.1:38133)
0925 07:24:37.216365 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.216369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.216380 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:24:37.216424 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.216427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.216443 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.216659 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:37.216741 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.217871 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.219179 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.219963 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.220397 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.220502 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.220527 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.220566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260925 07:24:37.220789 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.215851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:38133 (local address 127.0.0.1:58804)
0925 07:24:37.215971 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.216014 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.216086 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.216541 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.216541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.216543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:37.216578 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.216581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.217961 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.217962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.219044 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.219046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.220223 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.220223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.220272 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.220275 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.220305 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.220347 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260925 07:24:37.220866 24353 tablet_service.cc:1467] Processing CreateTablet for tablet 314a8fe6296c49ff9a714c982361a371 (DEFAULT_TABLE table=test-table [id=7701943d5710418ba1cf80fa4789fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:37.221290 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314a8fe6296c49ff9a714c982361a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:37.221475 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.216136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.4.47.2:43039)
0925 07:24:37.216729 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.216733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.216743 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:24:37.216779 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.216781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.216795 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.216881 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:37.217003 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.218033 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.219232 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.219963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.221233 (+ 1270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.221304 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.221328 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.221391 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":89,"threads_started":1}
I20260925 07:24:37.221552 24091 tablet_service.cc:1467] Processing CreateTablet for tablet 314a8fe6296c49ff9a714c982361a371 (DEFAULT_TABLE table=test-table [id=7701943d5710418ba1cf80fa4789fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:37.221909 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314a8fe6296c49ff9a714c982361a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:37.222110 24222 tablet_service.cc:1467] Processing CreateTablet for tablet 314a8fe6296c49ff9a714c982361a371 (DEFAULT_TABLE table=test-table [id=7701943d5710418ba1cf80fa4789fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:24:37.222450 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314a8fe6296c49ff9a714c982361a371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:37.224929 24433 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Bootstrap starting.
I20260925 07:24:37.225315 24435 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Bootstrap starting.
I20260925 07:24:37.225591 24434 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Bootstrap starting.
I20260925 07:24:37.225762 24433 tablet_bootstrap.cc:654] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:37.225965 24435 tablet_bootstrap.cc:654] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:37.226109 24433 log.cc:824] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:37.226298 24434 tablet_bootstrap.cc:654] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:37.226326 24435 log.cc:824] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:37.226598 24434 log.cc:824] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:37.226977 24433 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: No bootstrap required, opened a new log
I20260925 07:24:37.227053 24433 ts_tablet_manager.cc:1397] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:24:37.227131 24435 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: No bootstrap required, opened a new log
I20260925 07:24:37.227205 24435 ts_tablet_manager.cc:1397] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:24:37.229815 24435 raft_consensus.cc:348] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.229780 24433 raft_consensus.cc:348] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.229961 24435 raft_consensus.cc:374] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:37.229961 24433 raft_consensus.cc:374] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:37.229996 24435 raft_consensus.cc:729] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146836e64c104cdd9bfa77c519299d5c, State: Initialized, Role: FOLLOWER
I20260925 07:24:37.229996 24433 raft_consensus.cc:729] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf6c1c5cc034cd8baace0de1088f5c8, State: Initialized, Role: FOLLOWER
I20260925 07:24:37.230121 24435 consensus_queue.cc:260] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.230121 24433 consensus_queue.cc:260] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.230393 24433 ts_tablet_manager.cc:1428] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:24:37.230393 24435 ts_tablet_manager.cc:1428] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:24:37.230463 24157 heartbeater.cc:503] Master 127.4.47.62:42241 was elected leader, sending a full tablet report...
I20260925 07:24:37.230762 24419 heartbeater.cc:503] Master 127.4.47.62:42241 was elected leader, sending a full tablet report...
I20260925 07:24:37.231060 24434 tablet_bootstrap.cc:492] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: No bootstrap required, opened a new log
I20260925 07:24:37.231163 24434 ts_tablet_manager.cc:1397] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 07:24:37.233294 24434 raft_consensus.cc:348] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.233428 24434 raft_consensus.cc:374] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:37.233469 24434 raft_consensus.cc:729] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b25cfea905249cdbf1221a8efbb79fe, State: Initialized, Role: FOLLOWER
I20260925 07:24:37.233590 24434 consensus_queue.cc:260] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.233862 24434 ts_tablet_manager.cc:1428] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:24:37.233909 24288 heartbeater.cc:503] Master 127.4.47.62:42241 was elected leader, sending a full tablet report...
W20260925 07:24:37.327759 24289 tablet.cc:2369] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:37.445429 24158 tablet.cc:2369] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:24:37.446681 24420 tablet.cc:2369] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:24:37.466563 24441 raft_consensus.cc:482] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:24:37.466691 24441 raft_consensus.cc:504] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.467151 24441 leader_election.cc:288] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 146836e64c104cdd9bfa77c519299d5c (127.4.47.1:38133), ebf6c1c5cc034cd8baace0de1088f5c8 (127.4.47.3:43301)
I20260925 07:24:37.470005 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.467401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:49423 (local address 127.4.47.1:38133)
0925 07:24:37.467454 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.467457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.467492 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:24:37.467523 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.467524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.467535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.467586 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:37.467784 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.468257 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469231 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.469595 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469679 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.469690 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.469887 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.469932 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:24:37.470009 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.467313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:38133 (local address 127.4.47.2:49423)
0925 07:24:37.467388 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.467401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.467451 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.467645 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.467646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.467648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:37.467724 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.467725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.468371 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.468372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.469128 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.469130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.469692 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.469773 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.469775 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.469876 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.469932 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 07:24:37.470148 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.467534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.47.2:58251 (local address 127.4.47.3:43301)
0925 07:24:37.467583 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.467585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.467686 (+ 101us) server_negotiation.cc:407] Connection header received
0925 07:24:37.467739 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.467740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.467750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.467826 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:24:37.467992 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.468457 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469394 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.469757 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469849 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.469858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.470018 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.470055 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:24:37.470322 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.467437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.3:43301 (local address 127.4.47.2:58251)
0925 07:24:37.467618 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.467629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:24:37.467685 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:24:37.467832 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:24:37.467832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:24:37.467834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:24:37.467906 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.467908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.468552 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.468553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.469297 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:24:37.469299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.469856 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.469857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:24:37.469932 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.469935 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:24:37.470011 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:24:37.470049 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260925 07:24:37.470420 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314a8fe6296c49ff9a714c982361a371" candidate_uuid: "5b25cfea905249cdbf1221a8efbb79fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146836e64c104cdd9bfa77c519299d5c" is_pre_election: true
I20260925 07:24:37.470610 24111 raft_consensus.cc:2454] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b25cfea905249cdbf1221a8efbb79fe in term 0.
I20260925 07:24:37.470775 24373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314a8fe6296c49ff9a714c982361a371" candidate_uuid: "5b25cfea905249cdbf1221a8efbb79fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" is_pre_election: true
I20260925 07:24:37.470894 24178 leader_election.cc:302] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146836e64c104cdd9bfa77c519299d5c, 5b25cfea905249cdbf1221a8efbb79fe; no voters:
I20260925 07:24:37.470920 24373 raft_consensus.cc:2454] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b25cfea905249cdbf1221a8efbb79fe in term 0.
I20260925 07:24:37.471025 24441 raft_consensus.cc:2781] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:24:37.471112 24441 raft_consensus.cc:482] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:24:37.471144 24441 raft_consensus.cc:3037] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:37.471879 24441 raft_consensus.cc:504] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.472016 24441 leader_election.cc:288] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [CANDIDATE]: Term 1 election: Requested vote from peers 146836e64c104cdd9bfa77c519299d5c (127.4.47.1:38133), ebf6c1c5cc034cd8baace0de1088f5c8 (127.4.47.3:43301)
I20260925 07:24:37.472182 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314a8fe6296c49ff9a714c982361a371" candidate_uuid: "5b25cfea905249cdbf1221a8efbb79fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146836e64c104cdd9bfa77c519299d5c"
I20260925 07:24:37.472241 24373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314a8fe6296c49ff9a714c982361a371" candidate_uuid: "5b25cfea905249cdbf1221a8efbb79fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8"
I20260925 07:24:37.472276 24111 raft_consensus.cc:3037] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:37.472340 24373 raft_consensus.cc:3037] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:37.473213 24111 raft_consensus.cc:2454] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b25cfea905249cdbf1221a8efbb79fe in term 1.
I20260925 07:24:37.473213 24373 raft_consensus.cc:2454] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b25cfea905249cdbf1221a8efbb79fe in term 1.
I20260925 07:24:37.473421 24176 leader_election.cc:302] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b25cfea905249cdbf1221a8efbb79fe, ebf6c1c5cc034cd8baace0de1088f5c8; no voters:
I20260925 07:24:37.473556 24441 raft_consensus.cc:2781] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:37.473737 24441 raft_consensus.cc:686] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 1 LEADER]: Becoming Leader. State: Replica: 5b25cfea905249cdbf1221a8efbb79fe, State: Running, Role: LEADER
I20260925 07:24:37.473861 24441 consensus_queue.cc:237] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } }
I20260925 07:24:37.474799 23968 catalog_manager.cc:5495] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b25cfea905249cdbf1221a8efbb79fe (127.4.47.2). New cstate: current_term: 1 leader_uuid: "5b25cfea905249cdbf1221a8efbb79fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b25cfea905249cdbf1221a8efbb79fe" member_type: VOTER last_known_addr { host: "127.4.47.2" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 } health_report { overall_health: UNKNOWN } } }
I20260925 07:24:37.505227 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.502730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.4.47.2:43039)
0925 07:24:37.502816 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:37.502818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:37.502834 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:24:37.502872 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:37.502873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:37.502891 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:37.502973 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:24:37.503156 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.503632 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.504423 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:37.504620 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:37.504697 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:37.504877 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:24:37.504978 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:24:37.505075 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:37.505093 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 07:24:37.505110 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:24:37.507272 24111 raft_consensus.cc:1261] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 1 FOLLOWER]: Refusing update from remote peer 5b25cfea905249cdbf1221a8efbb79fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:37.507247 24373 raft_consensus.cc:1261] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 1 FOLLOWER]: Refusing update from remote peer 5b25cfea905249cdbf1221a8efbb79fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:24:37.507692 24443 consensus_queue.cc:1035] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf6c1c5cc034cd8baace0de1088f5c8" member_type: VOTER last_known_addr { host: "127.4.47.3" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:37.507835 24441 consensus_queue.cc:1035] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "146836e64c104cdd9bfa77c519299d5c" member_type: VOTER last_known_addr { host: "127.4.47.1" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:24:37.510057 24446 mvcc.cc:205] Tried to move back new op lower bound from 7333155133466320896 to 7333155133333676032. Current Snapshot: MvccSnapshot[applied={T|T < 7333155133466320896}]
I20260925 07:24:37.510612 24448 mvcc.cc:205] Tried to move back new op lower bound from 7333155133466320896 to 7333155133333676032. Current Snapshot: MvccSnapshot[applied={T|T < 7333155133466320896}]
I20260925 07:24:37.510768 24447 mvcc.cc:205] Tried to move back new op lower bound from 7333155133466320896 to 7333155133333676032. Current Snapshot: MvccSnapshot[applied={T|T < 7333155133466320896}]
I20260925 07:24:37.512555 23969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:37.512658 23969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:24:37.513475 23969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fab014d319a54ad8a1886b8fbebedd68: Sending DeleteTablet for 3 replicas of tablet 314a8fe6296c49ff9a714c982361a371
I20260925 07:24:37.513798 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet 314a8fe6296c49ff9a714c982361a371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:37 UTC) from {username='slave'} at 127.0.0.1:46890
I20260925 07:24:37.513814 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet 314a8fe6296c49ff9a714c982361a371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:37 UTC) from {username='slave'} at 127.0.0.1:34698
I20260925 07:24:37.513942 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet 314a8fe6296c49ff9a714c982361a371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:24:37 UTC) from {username='slave'} at 127.0.0.1:58804
I20260925 07:24:37.514034 24457 tablet_replica.cc:330] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: stopping tablet replica
I20260925 07:24:37.514135 24457 raft_consensus.cc:2229] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:37.514201 24458 tablet_replica.cc:330] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c: stopping tablet replica
I20260925 07:24:37.514254 24457 raft_consensus.cc:2258] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:37.514300 24458 raft_consensus.cc:2229] T 314a8fe6296c49ff9a714c982361a371 P 146836e64c104cdd9bfa77c519299d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:24:37.514452 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24029
I20260925 07:24:37.515661 24457 ts_tablet_manager.cc:1905] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:24:37.517594 24457 ts_tablet_manager.cc:1918] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:24:37.517673 24457 log.cc:1196] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Deleting WAL directory at /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790320837143632-4284-0/minicluster-data/ts-2/wal/wals/314a8fe6296c49ff9a714c982361a371
I20260925 07:24:37.517938 24457 ts_tablet_manager.cc:1939] T 314a8fe6296c49ff9a714c982361a371 P ebf6c1c5cc034cd8baace0de1088f5c8: Deleting consensus metadata
I20260925 07:24:37.518386 23954 catalog_manager.cc:4841] TS ebf6c1c5cc034cd8baace0de1088f5c8 (127.4.47.3:43301): tablet 314a8fe6296c49ff9a714c982361a371 (table test-table [id=7701943d5710418ba1cf80fa4789fb74]) successfully deleted
W20260925 07:24:37.520011 23956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.47.1:38133 (error 108)
I20260925 07:24:37.520218 24456 tablet_replica.cc:330] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe: stopping tablet replica
I20260925 07:24:37.520350 24456 raft_consensus.cc:2229] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:24:37.520488 24456 raft_consensus.cc:2258] T 314a8fe6296c49ff9a714c982361a371 P 5b25cfea905249cdbf1221a8efbb79fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:24:37.520669 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24160
I20260925 07:24:37.521970 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.521576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.1:38133 (local address 127.0.0.1:58808)
0925 07:24:37.521876 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.521931 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.1:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
W20260925 07:24:37.522149 23956 catalog_manager.cc:4584] TS 146836e64c104cdd9bfa77c519299d5c (127.4.47.1:38133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 314a8fe6296c49ff9a714c982361a371: Network error: Client connection negotiation failed: client connection to 127.4.47.1:38133: connect: Connection refused (error 111)
I20260925 07:24:37.525607 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24291
I20260925 07:24:37.525636 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:37.525513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.2:43039 (local address 127.0.0.1:46912)
0925 07:24:37.525571 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:24:37.525598 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.2:43039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:24:37.530230 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 23938
2026-09-25T07:24:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (928 ms)
[----------] 6 tests from EncryptionPolicyTest (8949 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:37 dist-test-slave-zw07 krb5kdc[24462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:37 dist-test-slave-zw07 krb5kdc[24462](info): set up 2 sockets
Sep 25 07:24:37 dist-test-slave-zw07 krb5kdc[24462](info): commencing operation
krb5kdc: starting...
W20260925 07:24:38.563043 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:38.687896 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:38.774531 24475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:38.774755 24475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:38.774780 24475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:38.775918 24475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:38.775959 24475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:38.775974 24475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:38.775986 24475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:38.776000 24475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:38.777426 24475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:38.777637 24475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:38.777820 24475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:38.782922 24481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:38.782922 24480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:38.782979 24483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:38.783273 24475 server_base.cc:1034] running on GCE node
I20260925 07:24:38.783550 24475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:38.783874 24475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:38.785050 24475 hybrid_clock.cc:648] HybridClock initialized: now 1790321078785025 us; error 44 us; skew 500 ppm
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:38.787869 24475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:38.788405 24475 webserver.cc:460] Webserver started at http://127.0.0.1:37665/ using document root <none> and password file <none>
I20260925 07:24:38.788666 24475 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:38.788738 24475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:38.788887 24475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:38.790076 24475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "caa64ee55ba6443881e6f05d6552049e"
format_stamp: "Formatted at 2026-09-25 07:24:38 on dist-test-slave-zw07"
server_key: "9a82b9d94ef49889f7b62c9b75ccdaec"
server_key_iv: "81437fd715b6fafe160878a89e5f52cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:38.790450 24475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "caa64ee55ba6443881e6f05d6552049e"
format_stamp: "Formatted at 2026-09-25 07:24:38 on dist-test-slave-zw07"
server_key: "9a82b9d94ef49889f7b62c9b75ccdaec"
server_key_iv: "81437fd715b6fafe160878a89e5f52cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:38.791966 24475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:38.793032 24490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:38.793291 24475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:38.793399 24475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "caa64ee55ba6443881e6f05d6552049e"
format_stamp: "Formatted at 2026-09-25 07:24:38 on dist-test-slave-zw07"
server_key: "9a82b9d94ef49889f7b62c9b75ccdaec"
server_key_iv: "81437fd715b6fafe160878a89e5f52cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:38.793481 24475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:38.818718 24475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:38.821193 24475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:38.821316 24475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:38.825623 24475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43643
I20260925 07:24:38.825692 24542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43643 every 8 connection(s)
I20260925 07:24:38.826049 24475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:38.827296 24543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:38.829841 24543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e: Bootstrap starting.
I20260925 07:24:38.830510 24543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:38.830838 24543 log.cc:824] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:38.831790 24543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e: No bootstrap required, opened a new log
I20260925 07:24:38.833549 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24475
I20260925 07:24:38.833714 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:38.833600 24543 raft_consensus.cc:348] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260925 07:24:38.833868 24543 raft_consensus.cc:374] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:38.833922 24543 raft_consensus.cc:729] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa64ee55ba6443881e6f05d6552049e, State: Initialized, Role: FOLLOWER
I20260925 07:24:38.833945 4284 external_mini_cluster.cc:1428] Setting key b0a893f364deb2a3dd9c06b15fe6f0c6
I20260925 07:24:38.834115 24543 consensus_queue.cc:260] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260925 07:24:38.834229 24543 raft_consensus.cc:388] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:38.834280 24543 raft_consensus.cc:482] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:38.834335 24543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:38.835629 24543 raft_consensus.cc:504] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260925 07:24:38.835768 24543 leader_election.cc:302] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caa64ee55ba6443881e6f05d6552049e; no voters:
I20260925 07:24:38.835961 24543 leader_election.cc:288] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:38.836025 24548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:38.836251 24548 raft_consensus.cc:686] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [term 1 LEADER]: Becoming Leader. State: Replica: caa64ee55ba6443881e6f05d6552049e, State: Running, Role: LEADER
I20260925 07:24:38.836267 24543 sys_catalog.cc:564] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:38.836347 24548 consensus_queue.cc:237] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260925 07:24:38.836706 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [sys.catalog]: SysCatalogTable state changed. Reason: New leader caa64ee55ba6443881e6f05d6552049e. Latest consensus state: current_term: 1 leader_uuid: "caa64ee55ba6443881e6f05d6552049e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260925 07:24:38.836809 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:38.836724 24549 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caa64ee55ba6443881e6f05d6552049e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caa64ee55ba6443881e6f05d6552049e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260925 07:24:38.836967 24549 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:38.837148 24555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:38.837925 24555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:38.839740 24555 catalog_manager.cc:1347] Generated new cluster ID: ce55a7e027104b24894170273aca2be2
I20260925 07:24:38.839797 24555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:24:38.847762 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:38.834596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36439 (local address 127.0.0.1:43643)
0925 07:24:38.834799 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:38.834810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:38.834850 (+ 40us) server_negotiation.cc:407] Connection header received
0925 07:24:38.835028 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:38.835034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:38.835148 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:38.835270 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:38.836243 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:38.837940 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:38.839173 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:38.839816 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:38.839906 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:38.843332 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:38.843337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:38.843337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:38.843351 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:38.845685 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:38.846193 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:38.846198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:38.846198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:38.846356 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:38.846629 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:38.846630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:38.846631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:38.847034 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:38.847285 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:38.847382 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:24:38.847503 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260925 07:24:38.848974 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:38.858992 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:38.850596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55985 (local address 127.0.0.1:43643)
0925 07:24:38.850641 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:38.850644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:38.850877 (+ 233us) server_negotiation.cc:407] Connection header received
0925 07:24:38.850914 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:38.850915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:38.850929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:38.851002 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:38.851680 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:38.852553 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:38.853686 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:38.854072 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:38.854152 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:38.855778 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:38.855780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:38.855780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:38.855788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:38.857614 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:38.858094 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:38.858096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:38.858096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:38.858165 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:38.858411 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:38.858412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:38.858413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:38.858563 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:38.858693 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:38.858809 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 07:24:38.858880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:24:38.880913 24555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:38.881587 24555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:38.891095 24555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caa64ee55ba6443881e6f05d6552049e: Generated new TSK 0
I20260925 07:24:38.891366 24555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:38.914314 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24475
W20260925 07:24:38.919288 24460 connection.cc:535] client connection to 127.0.0.1:43643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43643): Connection reset by peer (error 104)
2026-09-25T07:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24574](info): set up 2 sockets
Sep 25 07:24:38 dist-test-slave-zw07 krb5kdc[24574](info): commencing operation
krb5kdc: starting...
W20260925 07:24:39.959239 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:40 dist-test-slave-zw07 krb5kdc[24574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:40.082983 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:40.167835 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:40.168107 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:40.168146 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:40.169354 24587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:40.169430 24587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:40.169453 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:40.169472 24587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:40.169504 24587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:40.171015 24587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:40.171337 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:40.171584 24587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:40.176757 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:40.176751 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:40.176751 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:40.177448 24587 server_base.cc:1034] running on GCE node
I20260925 07:24:40.177717 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:40.178022 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:40.179181 24587 hybrid_clock.cc:648] HybridClock initialized: now 1790321080179168 us; error 30 us; skew 500 ppm
Sep 25 07:24:40 dist-test-slave-zw07 krb5kdc[24574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:40.181762 24587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:40.182251 24587 webserver.cc:460] Webserver started at http://127.0.0.1:32919/ using document root <none> and password file <none>
I20260925 07:24:40.182508 24587 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:40.182576 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:40.182720 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:40.183969 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "2e55cb07075d48ed88a18fc354155cb4"
format_stamp: "Formatted at 2026-09-25 07:24:40 on dist-test-slave-zw07"
server_key: "7931baf9ec2cdcb7157b8949ec991930"
server_key_iv: "16e175887d2a0dbdf069db5535ca2b0f"
server_key_version: "encryptionkey@0"
I20260925 07:24:40.184345 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "2e55cb07075d48ed88a18fc354155cb4"
format_stamp: "Formatted at 2026-09-25 07:24:40 on dist-test-slave-zw07"
server_key: "7931baf9ec2cdcb7157b8949ec991930"
server_key_iv: "16e175887d2a0dbdf069db5535ca2b0f"
server_key_version: "encryptionkey@0"
I20260925 07:24:40.185695 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:40.186517 24602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:40.186690 24587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:40.186795 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "2e55cb07075d48ed88a18fc354155cb4"
format_stamp: "Formatted at 2026-09-25 07:24:40 on dist-test-slave-zw07"
server_key: "7931baf9ec2cdcb7157b8949ec991930"
server_key_iv: "16e175887d2a0dbdf069db5535ca2b0f"
server_key_version: "encryptionkey@0"
I20260925 07:24:40.186869 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:40.209316 24587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:40.211973 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:40.212141 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:40.216228 24587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33925
I20260925 07:24:40.216293 24654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33925 every 8 connection(s)
I20260925 07:24:40.216670 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:40.217707 24655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:40.218492 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24587
I20260925 07:24:40.218616 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:40.218812 4284 external_mini_cluster.cc:1428] Setting key 531b90d3c606f69d3f51a363c6b3331a
I20260925 07:24:40.221107 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: Bootstrap starting.
I20260925 07:24:40.221827 24655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:40.222129 24655 log.cc:824] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:40.222901 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: No bootstrap required, opened a new log
I20260925 07:24:40.225057 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:40.219405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40533 (local address 127.0.0.1:33925)
0925 07:24:40.219619 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:40.219630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:40.219664 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:24:40.220217 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:40.220235 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:40.220303 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:40.220459 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:40.220640 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:40.222008 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:40.222832 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:40.223489 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:40.223607 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:40.224309 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:40.224314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:40.224314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:40.224542 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:40.224583 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:40.224598 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:40.224683 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:40.224771 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 07:24:40.224803 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260925 07:24:40.225517 24655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } }
I20260925 07:24:40.225680 24655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:40.225734 24655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e55cb07075d48ed88a18fc354155cb4, State: Initialized, Role: FOLLOWER
I20260925 07:24:40.225883 24655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } }
I20260925 07:24:40.225975 24655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:40.226017 24655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:40.226066 24655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:40.226214 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:40.226792 24655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } }
I20260925 07:24:40.226907 24655 leader_election.cc:302] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e55cb07075d48ed88a18fc354155cb4; no voters:
I20260925 07:24:40.227105 24655 leader_election.cc:288] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:40.227304 24661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:40.227371 24655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:40.227651 24661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 2e55cb07075d48ed88a18fc354155cb4, State: Running, Role: LEADER
I20260925 07:24:40.227828 24661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } }
I20260925 07:24:40.228278 24661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e55cb07075d48ed88a18fc354155cb4. Latest consensus state: current_term: 1 leader_uuid: "2e55cb07075d48ed88a18fc354155cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } } }
I20260925 07:24:40.228307 24660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e55cb07075d48ed88a18fc354155cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e55cb07075d48ed88a18fc354155cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33925 } } }
I20260925 07:24:40.228425 24661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:40.228442 24660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4 [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:40.229328 24679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:40.229382 24679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:40.229424 24676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:40.229676 24676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:40.231323 24676 catalog_manager.cc:1347] Generated new cluster ID: 1f228cc5756f4a47821a5fe4238eda9a
I20260925 07:24:40.231383 24676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:40.234943 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:40.229145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33165 (local address 127.0.0.1:33925)
0925 07:24:40.229180 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:40.229183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:40.231274 (+ 2091us) server_negotiation.cc:407] Connection header received
0925 07:24:40.231325 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:40.231326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:40.231340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:40.231410 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:40.231661 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:40.232414 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:40.233492 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:40.233822 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:40.233901 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:40.234682 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:40.234684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:40.234684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:40.234689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:40.234706 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:40.234711 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:40.234777 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:40.234852 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:24:40.234865 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 07:24:40.256014 24676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:40.256680 24676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:40.260529 24676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e55cb07075d48ed88a18fc354155cb4: Generated new TSK 0
I20260925 07:24:40.260752 24676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:40.267618 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24587
W20260925 07:24:40.271844 24572 connection.cc:535] client connection to 127.0.0.1:33925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33925): Connection reset by peer (error 104)
2026-09-25T07:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:40 dist-test-slave-zw07 krb5kdc[24686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:40 dist-test-slave-zw07 krb5kdc[24686](info): set up 2 sockets
Sep 25 07:24:40 dist-test-slave-zw07 krb5kdc[24686](info): commencing operation
krb5kdc: starting...
W20260925 07:24:41.315418 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:41 dist-test-slave-zw07 krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:41.442970 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:41.528817 24699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:41.529030 24699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:41.529055 24699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:41.530198 24699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:41.530239 24699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:41.530254 24699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:41.530267 24699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:41.530278 24699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:41.531755 24699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:41.531956 24699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:41.532140 24699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:41.537281 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:41.537423 24699 server_base.cc:1034] running on GCE node
W20260925 07:24:41.537278 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:41.537340 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:41.537771 24699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:41.538043 24699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:41.539170 24699 hybrid_clock.cc:648] HybridClock initialized: now 1790321081539155 us; error 30 us; skew 500 ppm
Sep 25 07:24:41 dist-test-slave-zw07 krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:41.541920 24699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:41.542462 24699 webserver.cc:460] Webserver started at http://127.0.0.1:44149/ using document root <none> and password file <none>
I20260925 07:24:41.542726 24699 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:41.542799 24699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:41.542949 24699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:41.544162 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "9763e36ce9614991ac6dccfb4a0e8d91"
format_stamp: "Formatted at 2026-09-25 07:24:41 on dist-test-slave-zw07"
server_key: "5d42a7d48ac74ba34eb032f10bb2d1e5"
server_key_iv: "7455b2e52ec9766318066ce07dd26d1d"
server_key_version: "encryptionkey@0"
I20260925 07:24:41.544569 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "9763e36ce9614991ac6dccfb4a0e8d91"
format_stamp: "Formatted at 2026-09-25 07:24:41 on dist-test-slave-zw07"
server_key: "5d42a7d48ac74ba34eb032f10bb2d1e5"
server_key_iv: "7455b2e52ec9766318066ce07dd26d1d"
server_key_version: "encryptionkey@0"
I20260925 07:24:41.545964 24699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:41.546890 24714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:41.547191 24699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:41.547300 24699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "9763e36ce9614991ac6dccfb4a0e8d91"
format_stamp: "Formatted at 2026-09-25 07:24:41 on dist-test-slave-zw07"
server_key: "5d42a7d48ac74ba34eb032f10bb2d1e5"
server_key_iv: "7455b2e52ec9766318066ce07dd26d1d"
server_key_version: "encryptionkey@0"
I20260925 07:24:41.547386 24699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:41.550742 24699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:41.553256 24699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:41.553411 24699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:41.557358 24699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46813
I20260925 07:24:41.557449 24766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46813 every 8 connection(s)
I20260925 07:24:41.557775 24699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:41.558040 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24699
I20260925 07:24:41.558168 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:41.558393 4284 external_mini_cluster.cc:1428] Setting key 77688dfea0ed6189649a18db2198fbcf
I20260925 07:24:41.559152 24767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:41.560930 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:41.558935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34083 (local address 127.0.0.1:46813)
0925 07:24:41.559157 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:41.559166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:41.559191 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:24:41.559671 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:41.559677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:41.559705 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:41.559821 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:41.560474 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:41.560477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:41.560478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:41.560481 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:41.560518 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:41.560529 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:41.560583 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:41.560674 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 07:24:41.560728 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260925 07:24:41.562069 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:41.562570 24767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91: Bootstrap starting.
I20260925 07:24:41.563390 24767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:41.563769 24767 log.cc:824] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:41.564543 24767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91: No bootstrap required, opened a new log
I20260925 07:24:41.564998 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:41.563818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60955 (local address 127.0.0.1:46813)
0925 07:24:41.563861 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:41.563863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:41.563960 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:24:41.564019 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:41.564019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:41.564024 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:41.564098 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:41.564786 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:41.564788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:41.564788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:41.564792 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:41.564807 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:41.564811 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:41.564867 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:41.564917 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 07:24:41.564932 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:24:41.566533 24767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } }
I20260925 07:24:41.566682 24767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:41.566705 24767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9763e36ce9614991ac6dccfb4a0e8d91, State: Initialized, Role: FOLLOWER
I20260925 07:24:41.566854 24767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } }
I20260925 07:24:41.566948 24767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:41.566995 24767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:41.567047 24767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:41.567651 24767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } }
I20260925 07:24:41.567766 24767 leader_election.cc:302] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9763e36ce9614991ac6dccfb4a0e8d91; no voters:
I20260925 07:24:41.567988 24767 leader_election.cc:288] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:41.568133 24777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:41.568239 24767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:41.568308 24777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [term 1 LEADER]: Becoming Leader. State: Replica: 9763e36ce9614991ac6dccfb4a0e8d91, State: Running, Role: LEADER
I20260925 07:24:41.568403 24777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } }
I20260925 07:24:41.568856 24779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9763e36ce9614991ac6dccfb4a0e8d91. Latest consensus state: current_term: 1 leader_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } } }
I20260925 07:24:41.568873 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9763e36ce9614991ac6dccfb4a0e8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46813 } } }
I20260925 07:24:41.568969 24779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:41.568979 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:41.569232 24786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:41.569804 24786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:41.571506 24786 catalog_manager.cc:1347] Generated new cluster ID: e944f341967a4644958e7a26901c58cb
I20260925 07:24:41.571574 24786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:41.579241 24786 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:41.579452 24786 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:41.588366 24786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9763e36ce9614991ac6dccfb4a0e8d91: Generated new TSK 0
I20260925 07:24:41.588600 24786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:41.596202 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24699
2026-09-25T07:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:41 dist-test-slave-zw07 krb5kdc[24798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:41 dist-test-slave-zw07 krb5kdc[24798](info): set up 2 sockets
Sep 25 07:24:41 dist-test-slave-zw07 krb5kdc[24798](info): commencing operation
krb5kdc: starting...
W20260925 07:24:42.631299 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:42.757661 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:42.841611 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:42.841851 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:42.841876 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:42.843221 24811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:42.843267 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:42.843282 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:42.843295 24811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:42.843307 24811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:42.844871 24811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:42.845171 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:42.845430 24811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:42.850723 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:42.850762 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:42.850755 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:42.850920 24811 server_base.cc:1034] running on GCE node
I20260925 07:24:42.851300 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:42.851573 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:42.852730 24811 hybrid_clock.cc:648] HybridClock initialized: now 1790321082852721 us; error 34 us; skew 500 ppm
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:42.855470 24811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:42.855986 24811 webserver.cc:460] Webserver started at http://127.0.0.1:37845/ using document root <none> and password file <none>
I20260925 07:24:42.856246 24811 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:42.856317 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:42.856467 24811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:42.857553 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "d73da0cd9faa43cdbb329bf4758c13a9"
format_stamp: "Formatted at 2026-09-25 07:24:42 on dist-test-slave-zw07"
server_key: "18d7638dfb634a60cb1c2abcb7801df5"
server_key_iv: "16e7a321a9d96d6b5887284c5fccd759"
server_key_version: "encryptionkey@0"
I20260925 07:24:42.857932 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "d73da0cd9faa43cdbb329bf4758c13a9"
format_stamp: "Formatted at 2026-09-25 07:24:42 on dist-test-slave-zw07"
server_key: "18d7638dfb634a60cb1c2abcb7801df5"
server_key_iv: "16e7a321a9d96d6b5887284c5fccd759"
server_key_version: "encryptionkey@0"
I20260925 07:24:42.859422 24811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:42.860306 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:42.860502 24811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:42.860610 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "d73da0cd9faa43cdbb329bf4758c13a9"
format_stamp: "Formatted at 2026-09-25 07:24:42 on dist-test-slave-zw07"
server_key: "18d7638dfb634a60cb1c2abcb7801df5"
server_key_iv: "16e7a321a9d96d6b5887284c5fccd759"
server_key_version: "encryptionkey@0"
I20260925 07:24:42.860692 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:42.878615 24811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:42.881589 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:42.881783 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:42.886075 24811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46309
I20260925 07:24:42.886150 24878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46309 every 8 connection(s)
I20260925 07:24:42.886526 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:42.887566 24879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:42.889920 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: Bootstrap starting.
I20260925 07:24:42.890532 24879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:42.890826 24879 log.cc:824] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:42.891614 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: No bootstrap required, opened a new log
I20260925 07:24:42.893345 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24811
I20260925 07:24:42.893337 24879 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } }
I20260925 07:24:42.893496 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:42.893522 24879 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:42.893592 24879 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73da0cd9faa43cdbb329bf4758c13a9, State: Initialized, Role: FOLLOWER
I20260925 07:24:42.893702 4284 external_mini_cluster.cc:1428] Setting key 32fd49a7d149604ae13600969daa37df
I20260925 07:24:42.893778 24879 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } }
I20260925 07:24:42.893868 24879 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:42.893908 24879 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:42.893961 24879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:42.894649 24879 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } }
I20260925 07:24:42.894783 24879 leader_election.cc:302] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73da0cd9faa43cdbb329bf4758c13a9; no voters:
I20260925 07:24:42.894942 24879 leader_election.cc:288] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:42.895004 24884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:42.895205 24884 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [term 1 LEADER]: Becoming Leader. State: Replica: d73da0cd9faa43cdbb329bf4758c13a9, State: Running, Role: LEADER
I20260925 07:24:42.895229 24879 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:42.895370 24884 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } }
I20260925 07:24:42.895746 24885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } } }
I20260925 07:24:42.895725 24884 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73da0cd9faa43cdbb329bf4758c13a9. Latest consensus state: current_term: 1 leader_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73da0cd9faa43cdbb329bf4758c13a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46309 } } }
I20260925 07:24:42.895898 24885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:42.895906 24884 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9 [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:42.897184 24899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:42.897243 24899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:42.897297 24893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:42.897482 24893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:42.899133 24893 catalog_manager.cc:1347] Generated new cluster ID: 0fff00b32d7f4f9794db804a9e93d0f6
I20260925 07:24:42.899210 24893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:24:42.905865 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:42.894300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49423 (local address 127.0.0.1:46309)
0925 07:24:42.894774 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:42.894784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:42.894819 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:24:42.894945 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:42.894953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:42.895030 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:42.896092 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:42.896208 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:42.897762 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:42.898850 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:42.899466 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:42.899561 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:42.902118 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:42.902122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:42.902123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:42.902135 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:42.904113 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:42.904548 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:42.904551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:42.904552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:42.904673 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:42.904908 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:42.904909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:42.904909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:42.905205 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:42.905344 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:42.905500 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 07:24:42.905640 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":356,"threads_started":1}
I20260925 07:24:42.907122 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:42.916131 24893 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:42.916847 24893 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:42.919328 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:42.908597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54755 (local address 127.0.0.1:46309)
0925 07:24:42.908636 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:42.908638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:42.911309 (+ 2671us) server_negotiation.cc:407] Connection header received
0925 07:24:42.911377 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:42.911378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:42.911389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:42.911450 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:42.912156 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:42.912901 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:42.913985 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:42.914398 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:42.914467 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:42.916030 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:42.916031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:42.916032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:42.916039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:42.917812 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:42.918277 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:42.918278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:42.918278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:42.918345 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:42.918661 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:42.918662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:42.918663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:42.918846 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:42.918951 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:42.919137 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 07:24:42.919238 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:24:42.922111 24893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73da0cd9faa43cdbb329bf4758c13a9: Generated new TSK 0
I20260925 07:24:42.922330 24893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:42.925360 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24811
2026-09-25T07:24:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24910](info): set up 2 sockets
Sep 25 07:24:42 dist-test-slave-zw07 krb5kdc[24910](info): commencing operation
krb5kdc: starting...
W20260925 07:24:43.979102 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:44 dist-test-slave-zw07 krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:44.102732 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:44.185441 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:44.185720 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:44.185761 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:44.186977 24923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:44.187088 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:44.187163 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:44.187189 24923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:44.187209 24923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:44.188757 24923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:44.188974 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:44.189154 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:44.194370 24931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:44.194378 24928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:44.194389 24929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:44.194590 24923 server_base.cc:1034] running on GCE node
I20260925 07:24:44.194875 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:44.195272 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:44.196460 24923 hybrid_clock.cc:648] HybridClock initialized: now 1790321084196435 us; error 61 us; skew 500 ppm
Sep 25 07:24:44 dist-test-slave-zw07 krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:44.206910 24923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:44.207526 24923 webserver.cc:460] Webserver started at http://127.0.0.1:34809/ using document root <none> and password file <none>
I20260925 07:24:44.207783 24923 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:44.207856 24923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:44.208020 24923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:44.209148 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "c1cf64b0bf27485a865e7f336fce118a"
format_stamp: "Formatted at 2026-09-25 07:24:44 on dist-test-slave-zw07"
server_key: "9372d7a0349420f74dcbf71e7a44e41e"
server_key_iv: "4cb1c5d7d4640e5322277f7ddcc326cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:44.209527 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "c1cf64b0bf27485a865e7f336fce118a"
format_stamp: "Formatted at 2026-09-25 07:24:44 on dist-test-slave-zw07"
server_key: "9372d7a0349420f74dcbf71e7a44e41e"
server_key_iv: "4cb1c5d7d4640e5322277f7ddcc326cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:44.210969 24923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:44.211988 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:44.212241 24923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:44.212368 24923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "c1cf64b0bf27485a865e7f336fce118a"
format_stamp: "Formatted at 2026-09-25 07:24:44 on dist-test-slave-zw07"
server_key: "9372d7a0349420f74dcbf71e7a44e41e"
server_key_iv: "4cb1c5d7d4640e5322277f7ddcc326cd"
server_key_version: "encryptionkey@0"
I20260925 07:24:44.212463 24923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:44.233908 24923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:44.236125 24923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:44.236303 24923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:44.240624 24923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41461
I20260925 07:24:44.240767 24990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41461 every 8 connection(s)
I20260925 07:24:44.241096 24923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:44.242136 24991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:44.244658 24991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a: Bootstrap starting.
I20260925 07:24:44.245436 24991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:44.245711 24991 log.cc:824] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:44.246469 24991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a: No bootstrap required, opened a new log
I20260925 07:24:44.248299 24991 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260925 07:24:44.248456 24991 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:44.248478 24991 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cf64b0bf27485a865e7f336fce118a, State: Initialized, Role: FOLLOWER
I20260925 07:24:44.248438 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 24923
I20260925 07:24:44.248603 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:44.248654 24991 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260925 07:24:44.248751 24991 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:44.248798 24991 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:44.248842 4284 external_mini_cluster.cc:1428] Setting key b958fd8a1ebe0add67e1dd34506ece34
I20260925 07:24:44.248853 24991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:44.249632 24991 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260925 07:24:44.249775 24991 leader_election.cc:302] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1cf64b0bf27485a865e7f336fce118a; no voters:
I20260925 07:24:44.249951 24991 leader_election.cc:288] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:44.250138 24997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:44.250211 24991 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:44.250290 24997 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [term 1 LEADER]: Becoming Leader. State: Replica: c1cf64b0bf27485a865e7f336fce118a, State: Running, Role: LEADER
I20260925 07:24:44.250385 24997 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260925 07:24:44.250768 24997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1cf64b0bf27485a865e7f336fce118a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260925 07:24:44.250854 24997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:44.251042 24997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1cf64b0bf27485a865e7f336fce118a. Latest consensus state: current_term: 1 leader_uuid: "c1cf64b0bf27485a865e7f336fce118a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1cf64b0bf27485a865e7f336fce118a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260925 07:24:44.251139 25005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:44.251178 24997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:44.251808 25005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:44.253788 25005 catalog_manager.cc:1347] Generated new cluster ID: 74d8ea43c4784737a51b1e514bfb039e
I20260925 07:24:44.253855 25005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:44.255568 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:44.249686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.0.0.1:41461)
0925 07:24:44.249877 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:44.249888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:44.249926 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:24:44.250063 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:44.250071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:44.250132 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:44.250574 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:44.250706 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:44.252257 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:44.253280 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:44.253887 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:44.254023 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:44.254760 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:44.254764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:44.254765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:44.255010 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:44.255051 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:44.255108 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:44.255182 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:44.255260 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 07:24:44.255334 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260925 07:24:44.256539 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:44.262948 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:44.257964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52925 (local address 127.0.0.1:41461)
0925 07:24:44.258000 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:44.258002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:44.259273 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 07:24:44.259339 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:44.259340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:44.259350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:44.259401 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:44.259551 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:44.260296 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:44.261440 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:44.261799 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:44.261888 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:44.262669 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:44.262670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:44.262670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:44.262676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:44.262693 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:44.262698 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:44.262751 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:44.262833 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:24:44.262862 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:24:44.272884 25005 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:44.273473 25005 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:44.276754 25005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1cf64b0bf27485a865e7f336fce118a: Generated new TSK 0
I20260925 07:24:44.276973 25005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:44.284705 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 24923
2026-09-25T07:24:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:44 dist-test-slave-zw07 krb5kdc[25022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:44 dist-test-slave-zw07 krb5kdc[25022](info): set up 2 sockets
Sep 25 07:24:44 dist-test-slave-zw07 krb5kdc[25022](info): commencing operation
krb5kdc: starting...
W20260925 07:24:45.363207 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:45 dist-test-slave-zw07 krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:45.488579 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:45.569748 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:45.569964 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:45.569988 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:45.571233 25035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:45.571274 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:45.571290 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:45.571301 25035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:45.571313 25035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:45.572830 25035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:45.573024 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:45.573284 25035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:45.578413 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:45.578548 25035 server_base.cc:1034] running on GCE node
W20260925 07:24:45.578438 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:45.578644 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:45.578934 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:45.579340 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:45.580529 25035 hybrid_clock.cc:648] HybridClock initialized: now 1790321085580527 us; error 44 us; skew 500 ppm
Sep 25 07:24:45 dist-test-slave-zw07 krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:45.583199 25035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:45.583719 25035 webserver.cc:460] Webserver started at http://127.0.0.1:35263/ using document root <none> and password file <none>
I20260925 07:24:45.583961 25035 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:45.584029 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:45.584183 25035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:45.585317 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "98977666f0e6422c97432811a41d2fa0"
format_stamp: "Formatted at 2026-09-25 07:24:45 on dist-test-slave-zw07"
server_key: "5b583102c145c8b7b5192c5fd7ea574f"
server_key_iv: "3cb01e333ce7acdda3b41949df5a2702"
server_key_version: "encryptionkey@0"
I20260925 07:24:45.585693 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "98977666f0e6422c97432811a41d2fa0"
format_stamp: "Formatted at 2026-09-25 07:24:45 on dist-test-slave-zw07"
server_key: "5b583102c145c8b7b5192c5fd7ea574f"
server_key_iv: "3cb01e333ce7acdda3b41949df5a2702"
server_key_version: "encryptionkey@0"
I20260925 07:24:45.587184 25035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:45.587978 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:45.588167 25035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:45.588277 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "98977666f0e6422c97432811a41d2fa0"
format_stamp: "Formatted at 2026-09-25 07:24:45 on dist-test-slave-zw07"
server_key: "5b583102c145c8b7b5192c5fd7ea574f"
server_key_iv: "3cb01e333ce7acdda3b41949df5a2702"
server_key_version: "encryptionkey@0"
I20260925 07:24:45.588366 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:45.591908 25035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:45.594295 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:45.594475 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:45.599130 25035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37539
I20260925 07:24:45.599226 25102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37539 every 8 connection(s)
I20260925 07:24:45.599629 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:45.600651 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:45.603099 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0: Bootstrap starting.
I20260925 07:24:45.603534 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25035
I20260925 07:24:45.603662 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:45.603749 25103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:45.603884 4284 external_mini_cluster.cc:1428] Setting key 71721b28eb6fe29d9f330675fdc07d65
I20260925 07:24:45.604117 25103 log.cc:824] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:45.604905 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0: No bootstrap required, opened a new log
I20260925 07:24:45.606462 25103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260925 07:24:45.606611 25103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:45.606635 25103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98977666f0e6422c97432811a41d2fa0, State: Initialized, Role: FOLLOWER
I20260925 07:24:45.606727 25103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260925 07:24:45.606777 25103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:45.606797 25103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:45.606823 25103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:45.607861 25103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260925 07:24:45.608000 25103 leader_election.cc:302] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98977666f0e6422c97432811a41d2fa0; no voters:
I20260925 07:24:45.608163 25103 leader_election.cc:288] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:45.608253 25108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:45.608431 25108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 98977666f0e6422c97432811a41d2fa0, State: Running, Role: LEADER
I20260925 07:24:45.608445 25103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:45.609030 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:45.604513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50377 (local address 127.0.0.1:37539)
0925 07:24:45.607212 (+ 2699us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:45.607222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:45.607262 (+ 40us) server_negotiation.cc:407] Connection header received
0925 07:24:45.607410 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:45.607419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:45.607457 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:45.607634 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:45.608389 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:45.608393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:45.608393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:45.608397 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:45.608441 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:45.608457 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:45.608768 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:45.608798 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:24:45.608840 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2641,"thread_start_us":80,"threads_started":1}
I20260925 07:24:45.608556 25108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260925 07:24:45.610033 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:45.610647 25110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98977666f0e6422c97432811a41d2fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260925 07:24:45.610728 25110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:45.610496 25109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98977666f0e6422c97432811a41d2fa0. Latest consensus state: current_term: 1 leader_uuid: "98977666f0e6422c97432811a41d2fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98977666f0e6422c97432811a41d2fa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260925 07:24:45.610874 25109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:45.611006 25123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:45.611558 25123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:45.612818 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:45.611810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43057 (local address 127.0.0.1:37539)
0925 07:24:45.611843 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:45.611844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:45.611852 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:24:45.611874 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:45.611875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:45.611880 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:45.611931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:45.612558 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:45.612559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:45.612559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:45.612564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:45.612578 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:45.612582 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:45.612649 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:45.612706 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 07:24:45.612721 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:24:45.613453 25123 catalog_manager.cc:1347] Generated new cluster ID: 00826b73b3a74b5bb98d1a2d1260a90d
I20260925 07:24:45.613514 25123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:45.624236 25123 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:45.624459 25123 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:45.632381 25123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98977666f0e6422c97432811a41d2fa0: Generated new TSK 0
I20260925 07:24:45.632616 25123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:45.637319 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25035
2026-09-25T07:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:45 dist-test-slave-zw07 krb5kdc[25134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:45 dist-test-slave-zw07 krb5kdc[25134](info): set up 2 sockets
Sep 25 07:24:45 dist-test-slave-zw07 krb5kdc[25134](info): commencing operation
krb5kdc: starting...
W20260925 07:24:46.671363 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:46 dist-test-slave-zw07 krb5kdc[25134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:46.798785 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:46.883651 25147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:46.883867 25147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:46.883894 25147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:46.885069 25147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:46.885118 25147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:46.885133 25147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:46.885144 25147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:46.885156 25147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:46.886662 25147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:46.886873 25147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:46.887102 25147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:46.891873 25152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:46.891870 25155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:46.891870 25153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:46.892256 25147 server_base.cc:1034] running on GCE node
I20260925 07:24:46.892481 25147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:46.892786 25147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:46.893937 25147 hybrid_clock.cc:648] HybridClock initialized: now 1790321086893919 us; error 29 us; skew 500 ppm
Sep 25 07:24:46 dist-test-slave-zw07 krb5kdc[25134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:46.896508 25147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:46.897013 25147 webserver.cc:460] Webserver started at http://127.0.0.1:34593/ using document root <none> and password file <none>
I20260925 07:24:46.897313 25147 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:46.897396 25147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:46.897544 25147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:46.898785 25147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "f555056bf1cb477db86133049c641f5a"
format_stamp: "Formatted at 2026-09-25 07:24:46 on dist-test-slave-zw07"
server_key: "916dd9ebcfab9762d9a4d35958612cc6"
server_key_iv: "e63975b984c6ab431d99aa7d8b11b5af"
server_key_version: "encryptionkey@0"
I20260925 07:24:46.899209 25147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "f555056bf1cb477db86133049c641f5a"
format_stamp: "Formatted at 2026-09-25 07:24:46 on dist-test-slave-zw07"
server_key: "916dd9ebcfab9762d9a4d35958612cc6"
server_key_iv: "e63975b984c6ab431d99aa7d8b11b5af"
server_key_version: "encryptionkey@0"
I20260925 07:24:46.900571 25147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:46.901363 25162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:46.901554 25147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:46.901647 25147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "f555056bf1cb477db86133049c641f5a"
format_stamp: "Formatted at 2026-09-25 07:24:46 on dist-test-slave-zw07"
server_key: "916dd9ebcfab9762d9a4d35958612cc6"
server_key_iv: "e63975b984c6ab431d99aa7d8b11b5af"
server_key_version: "encryptionkey@0"
I20260925 07:24:46.901733 25147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:46.911563 25147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:46.913975 25147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:46.914148 25147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:46.918391 25147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41325
I20260925 07:24:46.918426 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41325 every 8 connection(s)
I20260925 07:24:46.918825 25147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:46.919950 25215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:46.922330 25215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: Bootstrap starting.
I20260925 07:24:46.922964 25215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:46.923305 25215 log.cc:824] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:46.924052 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25147
I20260925 07:24:46.924168 25215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: No bootstrap required, opened a new log
I20260925 07:24:46.924186 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:46.924394 4284 external_mini_cluster.cc:1428] Setting key bb47f3c1e581bd48f38ef973724b06ec
I20260925 07:24:46.926733 25215 raft_consensus.cc:348] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260925 07:24:46.926910 25215 raft_consensus.cc:374] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:46.926950 25215 raft_consensus.cc:729] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f555056bf1cb477db86133049c641f5a, State: Initialized, Role: FOLLOWER
I20260925 07:24:46.927167 25215 consensus_queue.cc:260] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260925 07:24:46.927270 25215 raft_consensus.cc:388] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:46.927314 25215 raft_consensus.cc:482] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:46.927369 25215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:46.927981 25215 raft_consensus.cc:504] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260925 07:24:46.928102 25215 leader_election.cc:302] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f555056bf1cb477db86133049c641f5a; no voters:
I20260925 07:24:46.928277 25215 leader_election.cc:288] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:46.928354 25220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:46.928501 25220 raft_consensus.cc:686] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [term 1 LEADER]: Becoming Leader. State: Replica: f555056bf1cb477db86133049c641f5a, State: Running, Role: LEADER
I20260925 07:24:46.928545 25215 sys_catalog.cc:564] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:46.928593 25220 consensus_queue.cc:237] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260925 07:24:46.928923 25220 sys_catalog.cc:455] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f555056bf1cb477db86133049c641f5a. Latest consensus state: current_term: 1 leader_uuid: "f555056bf1cb477db86133049c641f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260925 07:24:46.929023 25220 sys_catalog.cc:458] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:46.928933 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f555056bf1cb477db86133049c641f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f555056bf1cb477db86133049c641f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260925 07:24:46.929142 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:46.930330 25235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:46.930374 25235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:46.930778 25226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:46.931000 25226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:24:46 dist-test-slave-zw07 krb5kdc[25134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:24:46.932871 25226 catalog_manager.cc:1347] Generated new cluster ID: 40bc9829e08c4675b27afbaa9685c0c1
I20260925 07:24:46.932969 25226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:46.937299 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:46.924951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:42313 (local address 127.0.0.1:41325)
0925 07:24:46.925138 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:46.925146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:46.925175 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:24:46.925601 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:46.925608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:46.925660 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:46.925777 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:46.926647 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:46.928052 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:46.929199 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:46.929795 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:46.929911 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:46.933238 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:46.933244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:46.933244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:46.933257 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:46.935407 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:46.935886 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:46.935890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:46.935890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:46.936013 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:46.936272 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:46.936273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:46.936273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:46.936616 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:46.936842 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:46.936945 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 07:24:46.937069 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260925 07:24:46.938107 25226 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:46.938421 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:46.938665 25226 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:46.946419 25226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f555056bf1cb477db86133049c641f5a: Generated new TSK 0
I20260925 07:24:46.946661 25226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:46.951311 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:46.943547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:41101 (local address 127.0.0.1:41325)
0925 07:24:46.943599 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:46.943601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:46.943774 (+ 173us) server_negotiation.cc:407] Connection header received
0925 07:24:46.943814 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:46.943815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:46.943827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:46.943892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:46.944569 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:46.945394 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:46.946729 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:46.947047 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:46.947181 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:46.948629 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:46.948630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:46.948631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:46.948637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:46.950057 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:46.950497 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:46.950498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:46.950498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:46.950552 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:46.950798 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:46.950798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:46.950799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:46.950897 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:46.950982 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:46.951170 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 07:24:46.951225 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:24:46.953398 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25147
W20260925 07:24:46.958177 25132 connection.cc:535] client connection to 127.0.0.1:41325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41325): Connection reset by peer (error 104)
2026-09-25T07:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:47 dist-test-slave-zw07 krb5kdc[25246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:47 dist-test-slave-zw07 krb5kdc[25246](info): set up 2 sockets
Sep 25 07:24:47 dist-test-slave-zw07 krb5kdc[25246](info): commencing operation
krb5kdc: starting...
W20260925 07:24:47.991281 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:48 dist-test-slave-zw07 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:48.114831 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:48.198158 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:48.198441 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:48.198486 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:48.199759 25259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:48.199880 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:48.199910 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:48.199944 25259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:48.199971 25259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:48.201650 25259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:48.201977 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:48.202236 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:48.207520 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:48.207509 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:48.207509 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:48.207590 25259 server_base.cc:1034] running on GCE node
I20260925 07:24:48.208031 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:48.208364 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:48.209537 25259 hybrid_clock.cc:648] HybridClock initialized: now 1790321088209523 us; error 49 us; skew 500 ppm
Sep 25 07:24:48 dist-test-slave-zw07 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:48.212365 25259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:48.212829 25259 webserver.cc:460] Webserver started at http://127.0.0.1:41145/ using document root <none> and password file <none>
I20260925 07:24:48.213097 25259 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:48.213169 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:48.213321 25259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:48.214443 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3"
format_stamp: "Formatted at 2026-09-25 07:24:48 on dist-test-slave-zw07"
server_key: "df022b940509edfb4e9f3cd50538c1be"
server_key_iv: "c8429a3768923ab943a0165038504df7"
server_key_version: "encryptionkey@0"
I20260925 07:24:48.214816 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3"
format_stamp: "Formatted at 2026-09-25 07:24:48 on dist-test-slave-zw07"
server_key: "df022b940509edfb4e9f3cd50538c1be"
server_key_iv: "c8429a3768923ab943a0165038504df7"
server_key_version: "encryptionkey@0"
I20260925 07:24:48.216338 25259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:48.217245 25274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:48.217445 25259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:48.217551 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3"
format_stamp: "Formatted at 2026-09-25 07:24:48 on dist-test-slave-zw07"
server_key: "df022b940509edfb4e9f3cd50538c1be"
server_key_iv: "c8429a3768923ab943a0165038504df7"
server_key_version: "encryptionkey@0"
I20260925 07:24:48.217639 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:48.237788 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:48.240104 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:48.240284 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:48.245491 25259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39377
I20260925 07:24:48.245648 25326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39377 every 8 connection(s)
I20260925 07:24:48.245973 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:48.247035 25327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:48.249529 25327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3: Bootstrap starting.
I20260925 07:24:48.249980 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25259
I20260925 07:24:48.250134 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:48.250237 25327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:48.250363 4284 external_mini_cluster.cc:1428] Setting key f52801be2f23c7d164b516ff2f12eb94
I20260925 07:24:48.250590 25327 log.cc:824] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:48.251470 25327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3: No bootstrap required, opened a new log
I20260925 07:24:48.253741 25327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } }
I20260925 07:24:48.253925 25327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:48.253957 25327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9435583ac2c54ef0b9fbe140ef8dcfd3, State: Initialized, Role: FOLLOWER
I20260925 07:24:48.254083 25327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } }
I20260925 07:24:48.254154 25327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:48.254184 25327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:48.254232 25327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:48.254855 25327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } }
I20260925 07:24:48.254971 25327 leader_election.cc:302] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9435583ac2c54ef0b9fbe140ef8dcfd3; no voters:
I20260925 07:24:48.255182 25327 leader_election.cc:288] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:48.255223 25332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:48.255379 25332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [term 1 LEADER]: Becoming Leader. State: Replica: 9435583ac2c54ef0b9fbe140ef8dcfd3, State: Running, Role: LEADER
I20260925 07:24:48.255502 25327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:48.255544 25332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } }
I20260925 07:24:48.256672 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:48.250958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:53025 (local address 127.0.0.1:39377)
0925 07:24:48.251153 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:48.251163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:48.251199 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:24:48.251814 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:48.251822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:48.251902 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:48.252029 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:48.252275 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:48.253170 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:48.254315 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:48.254770 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:48.254909 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:48.255756 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:48.255761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:48.255762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:48.255971 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:48.256012 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:48.256029 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:48.256132 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:48.256442 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 07:24:48.256473 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260925 07:24:48.256790 25334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9435583ac2c54ef0b9fbe140ef8dcfd3. Latest consensus state: current_term: 1 leader_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } } }
I20260925 07:24:48.256887 25334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:48.257077 25333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9435583ac2c54ef0b9fbe140ef8dcfd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39377 } } }
I20260925 07:24:48.257156 25333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:48.257185 25345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:48.257802 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:48.257833 25345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:48.259686 25345 catalog_manager.cc:1347] Generated new cluster ID: 68da37ee9f5e45499cd94b91350b74ca
I20260925 07:24:48.259742 25345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:48.262377 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:48.259258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:60261 (local address 127.0.0.1:39377)
0925 07:24:48.259299 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:48.259301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:48.259521 (+ 220us) server_negotiation.cc:407] Connection header received
0925 07:24:48.259553 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:48.259554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:48.259565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:48.259606 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:48.259811 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:48.260317 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:48.261017 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:48.261286 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:48.261379 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:48.262111 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:48.262113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:48.262113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:48.262118 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:48.262132 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:48.262137 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:48.262194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:48.262270 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:24:48.262279 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:24:48.279386 25345 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:48.280032 25345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:48.287046 25345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9435583ac2c54ef0b9fbe140ef8dcfd3: Generated new TSK 0
I20260925 07:24:48.287339 25345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:48.294202 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25259
W20260925 07:24:48.299152 25244 connection.cc:535] client connection to 127.0.0.1:39377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39377): Connection reset by peer (error 104)
2026-09-25T07:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:48 dist-test-slave-zw07 krb5kdc[25358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:48 dist-test-slave-zw07 krb5kdc[25358](info): set up 2 sockets
Sep 25 07:24:48 dist-test-slave-zw07 krb5kdc[25358](info): commencing operation
krb5kdc: starting...
W20260925 07:24:49.343278 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:49 dist-test-slave-zw07 krb5kdc[25358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:49.462416 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:49.545202 25371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:49.545493 25371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:49.545545 25371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:49.546763 25371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:49.546849 25371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:49.546875 25371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:49.546895 25371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:49.546921 25371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:49.548530 25371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:49.548851 25371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:49.549101 25371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:49.554318 25376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:49.554320 25377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:49.554438 25371 server_base.cc:1034] running on GCE node
W20260925 07:24:49.554529 25379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:49.554800 25371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:49.555200 25371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:49.556372 25371 hybrid_clock.cc:648] HybridClock initialized: now 1790321089556361 us; error 41 us; skew 500 ppm
Sep 25 07:24:49 dist-test-slave-zw07 krb5kdc[25358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:49.567754 25371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:49.568295 25371 webserver.cc:460] Webserver started at http://127.0.0.1:36571/ using document root <none> and password file <none>
I20260925 07:24:49.568511 25371 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:49.568555 25371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:49.568652 25371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:49.569730 25371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "aa68036b806e45ba82657eeac037ce6a"
format_stamp: "Formatted at 2026-09-25 07:24:49 on dist-test-slave-zw07"
server_key: "4c4be44bfcc917dffd1a4ad2ada9c0d8"
server_key_iv: "e2377a5434684572054585d556b69d61"
server_key_version: "encryptionkey@0"
I20260925 07:24:49.570062 25371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "aa68036b806e45ba82657eeac037ce6a"
format_stamp: "Formatted at 2026-09-25 07:24:49 on dist-test-slave-zw07"
server_key: "4c4be44bfcc917dffd1a4ad2ada9c0d8"
server_key_iv: "e2377a5434684572054585d556b69d61"
server_key_version: "encryptionkey@0"
I20260925 07:24:49.571516 25371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:49.572357 25386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:49.572558 25371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:49.572641 25371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "aa68036b806e45ba82657eeac037ce6a"
format_stamp: "Formatted at 2026-09-25 07:24:49 on dist-test-slave-zw07"
server_key: "4c4be44bfcc917dffd1a4ad2ada9c0d8"
server_key_iv: "e2377a5434684572054585d556b69d61"
server_key_version: "encryptionkey@0"
I20260925 07:24:49.572702 25371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:49.576149 25371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:49.578239 25371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:49.578351 25371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:49.582633 25371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35889
I20260925 07:24:49.582696 25438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35889 every 8 connection(s)
I20260925 07:24:49.583042 25371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:49.584179 25439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:49.586740 25439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a: Bootstrap starting.
I20260925 07:24:49.587441 25439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:49.587455 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25371
I20260925 07:24:49.587543 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:49.587738 4284 external_mini_cluster.cc:1428] Setting key 6661ce61d6e33df5d73060f88783eaf2
I20260925 07:24:49.587857 25439 log.cc:824] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:49.588845 25439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a: No bootstrap required, opened a new log
I20260925 07:24:49.590454 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:49.588371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:56603 (local address 127.0.0.1:35889)
0925 07:24:49.588564 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:49.588573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:49.588602 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:24:49.589144 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:49.589152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:49.589181 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:49.589306 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:49.589987 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:49.589990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:49.589990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:49.589994 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:49.590030 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:49.590041 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:49.590108 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:49.590188 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 07:24:49.590224 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 07:24:49.591564 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:49.591732 25439 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 07:24:49.591925 25439 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:49.591969 25439 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa68036b806e45ba82657eeac037ce6a, State: Initialized, Role: FOLLOWER
I20260925 07:24:49.592110 25439 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 07:24:49.592214 25439 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:49.592250 25439 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:49.592301 25439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:49.593223 25439 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 07:24:49.593355 25439 leader_election.cc:302] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa68036b806e45ba82657eeac037ce6a; no voters:
I20260925 07:24:49.593565 25439 leader_election.cc:288] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:49.593592 25449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:49.593784 25449 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [term 1 LEADER]: Becoming Leader. State: Replica: aa68036b806e45ba82657eeac037ce6a, State: Running, Role: LEADER
I20260925 07:24:49.593842 25439 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:49.593888 25449 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } }
I20260925 07:24:49.594231 25449 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa68036b806e45ba82657eeac037ce6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } } }
I20260925 07:24:49.594374 25449 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:49.594575 25449 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa68036b806e45ba82657eeac037ce6a. Latest consensus state: current_term: 1 leader_uuid: "aa68036b806e45ba82657eeac037ce6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa68036b806e45ba82657eeac037ce6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35889 } } }
I20260925 07:24:49.594655 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:49.593287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:59837 (local address 127.0.0.1:35889)
0925 07:24:49.593349 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:49.593351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:49.593489 (+ 138us) server_negotiation.cc:407] Connection header received
0925 07:24:49.593566 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:49.593567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:49.593572 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:49.593632 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:49.594375 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:49.594376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:49.594377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:49.594381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:49.594397 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:49.594401 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:49.594434 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:49.594555 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 07:24:49.594570 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:24:49.594717 25449 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:49.595185 25456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:49.595733 25456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:49.597329 25456 catalog_manager.cc:1347] Generated new cluster ID: 201840bcc71e4925945998bdc50e2179
I20260925 07:24:49.597407 25456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:49.622931 25456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:49.623201 25456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:49.626421 25456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa68036b806e45ba82657eeac037ce6a: Generated new TSK 0
I20260925 07:24:49.626631 25456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:49.632380 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25371
2026-09-25T07:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:49 dist-test-slave-zw07 krb5kdc[25470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:49 dist-test-slave-zw07 krb5kdc[25470](info): set up 2 sockets
Sep 25 07:24:49 dist-test-slave-zw07 krb5kdc[25470](info): commencing operation
krb5kdc: starting...
W20260925 07:24:50.659313 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:50 dist-test-slave-zw07 krb5kdc[25470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:50.787238 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:50.870364 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:50.870648 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:50.870689 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:50.872007 25483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:50.872087 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:50.872121 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:50.872153 25483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:50.872177 25483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:50.873891 25483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:50.874200 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:50.874445 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:50.879505 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:50.879527 25491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:50.879566 25483 server_base.cc:1034] running on GCE node
W20260925 07:24:50.879518 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:50.880010 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:50.880314 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:50.881484 25483 hybrid_clock.cc:648] HybridClock initialized: now 1790321090881471 us; error 36 us; skew 500 ppm
Sep 25 07:24:50 dist-test-slave-zw07 krb5kdc[25470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:50.884272 25483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:50.884771 25483 webserver.cc:460] Webserver started at http://127.0.0.1:44845/ using document root <none> and password file <none>
I20260925 07:24:50.885066 25483 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:50.885139 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:50.885285 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:50.886340 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "30b88522a8484ee0979aac3f0818d7bc"
format_stamp: "Formatted at 2026-09-25 07:24:50 on dist-test-slave-zw07"
server_key: "d7d5b5bb1f0c82b327c2b1616f07a5dd"
server_key_iv: "603bfb92af34bfdc0f3cec34473b606c"
server_key_version: "encryptionkey@0"
I20260925 07:24:50.886725 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "30b88522a8484ee0979aac3f0818d7bc"
format_stamp: "Formatted at 2026-09-25 07:24:50 on dist-test-slave-zw07"
server_key: "d7d5b5bb1f0c82b327c2b1616f07a5dd"
server_key_iv: "603bfb92af34bfdc0f3cec34473b606c"
server_key_version: "encryptionkey@0"
I20260925 07:24:50.888147 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:50.888903 25498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:50.889106 25483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:24:50.889214 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "30b88522a8484ee0979aac3f0818d7bc"
format_stamp: "Formatted at 2026-09-25 07:24:50 on dist-test-slave-zw07"
server_key: "d7d5b5bb1f0c82b327c2b1616f07a5dd"
server_key_iv: "603bfb92af34bfdc0f3cec34473b606c"
server_key_version: "encryptionkey@0"
I20260925 07:24:50.889288 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:50.919279 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:50.921527 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:50.921703 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:50.925961 25483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41381
I20260925 07:24:50.925999 25550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41381 every 8 connection(s)
I20260925 07:24:50.926381 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:50.927534 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:50.929924 25551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc: Bootstrap starting.
I20260925 07:24:50.930554 25551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:50.930866 25551 log.cc:824] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:50.931694 25551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc: No bootstrap required, opened a new log
I20260925 07:24:50.932899 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25483
I20260925 07:24:50.933027 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:50.933211 4284 external_mini_cluster.cc:1428] Setting key fdff9f913526a8990de89b4b452d8ff7
I20260925 07:24:50.933938 25551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260925 07:24:50.934100 25551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:50.934125 25551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b88522a8484ee0979aac3f0818d7bc, State: Initialized, Role: FOLLOWER
I20260925 07:24:50.934285 25551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260925 07:24:50.934376 25551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:50.934410 25551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:50.934455 25551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:50.935631 25551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260925 07:24:50.935724 25551 leader_election.cc:302] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b88522a8484ee0979aac3f0818d7bc; no voters:
I20260925 07:24:50.935952 25551 leader_election.cc:288] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:50.935973 25556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:50.936090 25556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [term 1 LEADER]: Becoming Leader. State: Replica: 30b88522a8484ee0979aac3f0818d7bc, State: Running, Role: LEADER
I20260925 07:24:50.936228 25551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:50.936236 25556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260925 07:24:50.936764 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b88522a8484ee0979aac3f0818d7bc. Latest consensus state: current_term: 1 leader_uuid: "30b88522a8484ee0979aac3f0818d7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } } }
I20260925 07:24:50.936909 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:50.937580 25557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b88522a8484ee0979aac3f0818d7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b88522a8484ee0979aac3f0818d7bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } } }
I20260925 07:24:50.937649 25557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:50.937731 25565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:50.938400 25565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:50.940286 25565 catalog_manager.cc:1347] Generated new cluster ID: d5b3fcdc5be94bfaa8706389cc9c5b0c
I20260925 07:24:50.940337 25565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:24:50 dist-test-slave-zw07 krb5kdc[25470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:24:50.945287 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:50.933772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:57155 (local address 127.0.0.1:41381)
0925 07:24:50.933984 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:50.933993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:50.934019 (+ 26us) server_negotiation.cc:407] Connection header received
0925 07:24:50.934141 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:50.934149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:50.934219 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:50.934395 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:50.935265 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:50.936437 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:50.937596 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:50.938198 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:50.938323 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:50.941225 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:50.941230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:50.941231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:50.941244 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:50.943352 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:50.943798 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:50.943801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:50.943801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:50.943948 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:50.944182 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:50.944184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:50.944184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:50.944536 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:50.944721 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:50.944811 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 07:24:50.945055 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260925 07:24:50.946413 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:50.954661 25565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:50.955346 25565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:50.959165 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:50.950924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:55003 (local address 127.0.0.1:41381)
0925 07:24:50.950972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:50.950974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:50.951111 (+ 137us) server_negotiation.cc:407] Connection header received
0925 07:24:50.951161 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:50.951162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:50.951173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:50.951239 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:50.951863 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:50.952654 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:50.953675 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:50.954020 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:50.954119 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:50.956098 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:50.956100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:50.956100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:50.956108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:50.957801 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:50.958236 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:50.958237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:50.958238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:50.958300 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:50.958530 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:50.958530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:50.958531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:50.958668 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:50.958775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:50.958926 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 07:24:50.959002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:24:50.961016 25565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b88522a8484ee0979aac3f0818d7bc: Generated new TSK 0
I20260925 07:24:50.961213 25565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:50.966161 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25483
W20260925 07:24:50.970456 25468 connection.cc:535] client connection to 127.0.0.1:41381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41381): Connection reset by peer (error 104)
2026-09-25T07:24:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:51 dist-test-slave-zw07 krb5kdc[25582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:51 dist-test-slave-zw07 krb5kdc[25582](info): set up 2 sockets
Sep 25 07:24:51 dist-test-slave-zw07 krb5kdc[25582](info): commencing operation
krb5kdc: starting...
W20260925 07:24:52.019551 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:52 dist-test-slave-zw07 krb5kdc[25582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:52.140241 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:52.224292 25595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:52.224522 25595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:52.224545 25595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:52.225798 25595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:52.225871 25595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:52.225914 25595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:52.225943 25595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:52.225965 25595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:52.227536 25595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:52.227798 25595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:52.228042 25595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:52.233031 25600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:52.233038 25601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:52.233045 25603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:52.233268 25595 server_base.cc:1034] running on GCE node
I20260925 07:24:52.233556 25595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:52.233853 25595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:52.235037 25595 hybrid_clock.cc:648] HybridClock initialized: now 1790321092235023 us; error 45 us; skew 500 ppm
Sep 25 07:24:52 dist-test-slave-zw07 krb5kdc[25582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:52.237603 25595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:52.238011 25595 webserver.cc:460] Webserver started at http://127.0.0.1:45167/ using document root <none> and password file <none>
I20260925 07:24:52.238193 25595 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:52.238242 25595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:52.238335 25595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:52.239468 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "3179f09d25b44fff9ce979faf9a46a26"
format_stamp: "Formatted at 2026-09-25 07:24:52 on dist-test-slave-zw07"
server_key: "669d558795bd1e68653cba25166e74d0"
server_key_iv: "dccd485eaee9f7ccf388d8505478c6d0"
server_key_version: "encryptionkey@0"
I20260925 07:24:52.239792 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "3179f09d25b44fff9ce979faf9a46a26"
format_stamp: "Formatted at 2026-09-25 07:24:52 on dist-test-slave-zw07"
server_key: "669d558795bd1e68653cba25166e74d0"
server_key_iv: "dccd485eaee9f7ccf388d8505478c6d0"
server_key_version: "encryptionkey@0"
I20260925 07:24:52.241096 25595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:52.241799 25610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:52.242007 25595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:24:52.242116 25595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "3179f09d25b44fff9ce979faf9a46a26"
format_stamp: "Formatted at 2026-09-25 07:24:52 on dist-test-slave-zw07"
server_key: "669d558795bd1e68653cba25166e74d0"
server_key_iv: "dccd485eaee9f7ccf388d8505478c6d0"
server_key_version: "encryptionkey@0"
I20260925 07:24:52.242192 25595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:52.251590 25595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:52.253533 25595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:52.253714 25595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:52.257934 25595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37973
I20260925 07:24:52.257970 25662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37973 every 8 connection(s)
I20260925 07:24:52.258471 25595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:52.259730 25663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:52.262503 25663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: Bootstrap starting.
I20260925 07:24:52.263186 25663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:52.263466 25663 log.cc:824] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:52.264236 25663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: No bootstrap required, opened a new log
I20260925 07:24:52.265422 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25595
I20260925 07:24:52.265580 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:52.265842 4284 external_mini_cluster.cc:1428] Setting key 4cb77fadbf9734424f16900f3c445efa
I20260925 07:24:52.265985 25663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260925 07:24:52.266145 25663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:52.266191 25663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3179f09d25b44fff9ce979faf9a46a26, State: Initialized, Role: FOLLOWER
I20260925 07:24:52.266374 25663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260925 07:24:52.266479 25663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:52.266537 25663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:52.266589 25663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:52.267181 25663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260925 07:24:52.267297 25663 leader_election.cc:302] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3179f09d25b44fff9ce979faf9a46a26; no voters:
I20260925 07:24:52.267460 25663 leader_election.cc:288] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:52.267481 25668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:52.267588 25668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [term 1 LEADER]: Becoming Leader. State: Replica: 3179f09d25b44fff9ce979faf9a46a26, State: Running, Role: LEADER
I20260925 07:24:52.267709 25663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:52.267717 25668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } }
I20260925 07:24:52.268088 25668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3179f09d25b44fff9ce979faf9a46a26. Latest consensus state: current_term: 1 leader_uuid: "3179f09d25b44fff9ce979faf9a46a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } } }
I20260925 07:24:52.268102 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3179f09d25b44fff9ce979faf9a46a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3179f09d25b44fff9ce979faf9a46a26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37973 } } }
I20260925 07:24:52.268210 25668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:52.268440 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26 [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:52.269621 25684 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:52.269680 25684 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:52.269724 25675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:52.269886 25675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:52.271466 25675 catalog_manager.cc:1347] Generated new cluster ID: 327bf7a0b5374e9d901c4409cc0b45b7
I20260925 07:24:52.271530 25675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:52.272220 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:52.266569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:42149 (local address 127.0.0.1:37973)
0925 07:24:52.266749 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:52.266765 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:52.266790 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:24:52.267169 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:52.267177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:52.267229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:52.267378 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:52.267606 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:52.268995 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:52.270041 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:52.270485 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:52.270597 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:52.271544 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:52.271548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:52.271548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:52.271729 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:52.271781 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:52.271792 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:52.271887 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:52.271979 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 07:24:52.272007 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260925 07:24:52.273149 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:52.277552 25675 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:52.278079 25675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:52.279691 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:52.274520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:54695 (local address 127.0.0.1:37973)
0925 07:24:52.274559 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:52.274561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:52.274703 (+ 142us) server_negotiation.cc:407] Connection header received
0925 07:24:52.274780 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:52.274782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:52.274792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:52.274847 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:52.275014 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:52.275673 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:52.276429 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:52.276764 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:52.276951 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:52.279375 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:52.279376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:52.279377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:52.279382 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:52.279399 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:52.279403 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:52.279476 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:52.279601 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 07:24:52.279617 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:24:52.290820 25675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3179f09d25b44fff9ce979faf9a46a26: Generated new TSK 0
I20260925 07:24:52.291060 25675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:52.295837 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25595
W20260925 07:24:52.300637 25580 connection.cc:535] client connection to 127.0.0.1:37973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37973): Connection reset by peer (error 104)
2026-09-25T07:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:52 dist-test-slave-zw07 krb5kdc[25694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:52 dist-test-slave-zw07 krb5kdc[25694](info): set up 2 sockets
Sep 25 07:24:52 dist-test-slave-zw07 krb5kdc[25694](info): commencing operation
krb5kdc: starting...
W20260925 07:24:53.331358 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:53 dist-test-slave-zw07 krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:24:53.449896 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:53.533027 25707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:53.533286 25707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:53.533334 25707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:53.534575 25707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:53.534649 25707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:53.534677 25707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:53.534706 25707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:53.534742 25707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:53.536406 25707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:53.536684 25707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:53.536923 25707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:53.541926 25715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:53.542011 25712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:53.542027 25713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:53.542124 25707 server_base.cc:1034] running on GCE node
I20260925 07:24:53.542428 25707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:53.542747 25707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:53.543915 25707 hybrid_clock.cc:648] HybridClock initialized: now 1790321093543899 us; error 46 us; skew 500 ppm
Sep 25 07:24:53 dist-test-slave-zw07 krb5kdc[25694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:53.546476 25707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:24:53.546950 25707 webserver.cc:460] Webserver started at http://127.0.0.1:40831/ using document root <none> and password file <none>
I20260925 07:24:53.547225 25707 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:53.547276 25707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:53.547408 25707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:53.548512 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "4daed0ebfa254eadae00a1afdc5af659"
format_stamp: "Formatted at 2026-09-25 07:24:53 on dist-test-slave-zw07"
server_key: "dea8f3fda9c5489a5edfbca93aaf476d"
server_key_iv: "f0818e2ceb37ca0ef72f4398be89ea63"
server_key_version: "encryptionkey@0"
I20260925 07:24:53.548911 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "4daed0ebfa254eadae00a1afdc5af659"
format_stamp: "Formatted at 2026-09-25 07:24:53 on dist-test-slave-zw07"
server_key: "dea8f3fda9c5489a5edfbca93aaf476d"
server_key_iv: "f0818e2ceb37ca0ef72f4398be89ea63"
server_key_version: "encryptionkey@0"
I20260925 07:24:53.550323 25707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:53.551178 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:53.551370 25707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:53.551481 25707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "4daed0ebfa254eadae00a1afdc5af659"
format_stamp: "Formatted at 2026-09-25 07:24:53 on dist-test-slave-zw07"
server_key: "dea8f3fda9c5489a5edfbca93aaf476d"
server_key_iv: "f0818e2ceb37ca0ef72f4398be89ea63"
server_key_version: "encryptionkey@0"
I20260925 07:24:53.551542 25707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:53.555349 25707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:53.557382 25707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:53.557499 25707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:53.561405 25707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44955
I20260925 07:24:53.561434 25774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44955 every 8 connection(s)
I20260925 07:24:53.561787 25707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:53.562659 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:53.564988 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25707
I20260925 07:24:53.565007 25775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659: Bootstrap starting.
I20260925 07:24:53.565091 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:53.565310 4284 external_mini_cluster.cc:1428] Setting key f482d9d783ef62b074f5968310856d47
I20260925 07:24:53.565759 25775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:53.566174 25775 log.cc:824] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:53.566951 25775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659: No bootstrap required, opened a new log
I20260925 07:24:53.569397 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:53.565868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:45309 (local address 127.0.0.1:44955)
0925 07:24:53.566090 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:53.566106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:53.566144 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:24:53.566718 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:53.566727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:53.566763 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:53.566927 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:53.568885 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:53.568890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:53.568890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:53.568896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:53.568952 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:53.568971 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:53.569082 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:53.569121 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 07:24:53.569174 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260925 07:24:53.569547 25775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260925 07:24:53.569687 25775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:53.569710 25775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4daed0ebfa254eadae00a1afdc5af659, State: Initialized, Role: FOLLOWER
I20260925 07:24:53.569841 25775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260925 07:24:53.569929 25775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:53.569962 25775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:53.570009 25775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:53.570570 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:53.570689 25775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260925 07:24:53.570814 25775 leader_election.cc:302] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4daed0ebfa254eadae00a1afdc5af659; no voters:
I20260925 07:24:53.571019 25775 leader_election.cc:288] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:53.571044 25780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:53.571233 25780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [term 1 LEADER]: Becoming Leader. State: Replica: 4daed0ebfa254eadae00a1afdc5af659, State: Running, Role: LEADER
I20260925 07:24:53.571302 25775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:53.571372 25780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260925 07:24:53.571691 25780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4daed0ebfa254eadae00a1afdc5af659. Latest consensus state: current_term: 1 leader_uuid: "4daed0ebfa254eadae00a1afdc5af659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260925 07:24:53.571703 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4daed0ebfa254eadae00a1afdc5af659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4daed0ebfa254eadae00a1afdc5af659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260925 07:24:53.571838 25780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:53.571877 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:53.572176 25793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:53.572711 25793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:53.574240 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:53.572304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:54993 (local address 127.0.0.1:44955)
0925 07:24:53.572340 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:53.572341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:53.572935 (+ 594us) server_negotiation.cc:407] Connection header received
0925 07:24:53.573245 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:53.573245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:53.573251 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:53.573304 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:53.573994 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:53.573996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:53.573996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:53.574002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:53.574020 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:53.574024 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:53.574087 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:53.574151 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 07:24:53.574165 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:24:53.574662 25793 catalog_manager.cc:1347] Generated new cluster ID: 4f1e0a7a5f99407e9d4609ab90d3a57f
I20260925 07:24:53.574712 25793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:53.586473 25793 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:53.586776 25793 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:53.593132 25793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4daed0ebfa254eadae00a1afdc5af659: Generated new TSK 0
I20260925 07:24:53.593385 25793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:53.604171 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25707
2026-09-25T07:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:53 dist-test-slave-zw07 krb5kdc[25806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:53 dist-test-slave-zw07 krb5kdc[25806](info): set up 2 sockets
Sep 25 07:24:53 dist-test-slave-zw07 krb5kdc[25806](info): commencing operation
krb5kdc: starting...
W20260925 07:24:54.639313 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:54.766851 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:35935
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:54.848948 25819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:54.849229 25819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:54.849267 25819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:54.850441 25819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:54.850520 25819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:54.850543 25819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:54.850564 25819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:54.850593 25819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:54.852125 25819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:35935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:54.852422 25819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:54.852662 25819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:54.857844 25825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:54.857844 25824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:54.857898 25819 server_base.cc:1034] running on GCE node
W20260925 07:24:54.857839 25827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:54.858398 25819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:54.858654 25819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:54.859797 25819 hybrid_clock.cc:648] HybridClock initialized: now 1790321094859776 us; error 41 us; skew 500 ppm
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321094, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:54.866725 25819 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:54.867291 25819 webserver.cc:460] Webserver started at http://127.4.47.62:33211/ using document root <none> and password file <none>
I20260925 07:24:54.867542 25819 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:54.867587 25819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:54.867753 25819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:54.868851 25819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "73259d9904864c309923ad9dc434d19c"
format_stamp: "Formatted at 2026-09-25 07:24:54 on dist-test-slave-zw07"
server_key: "5c7380fd26345d5ef42e874565ac54bb"
server_key_iv: "aa4be8a11c72d1421e6b372d58f309d4"
server_key_version: "encryptionkey@0"
I20260925 07:24:54.869232 25819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "73259d9904864c309923ad9dc434d19c"
format_stamp: "Formatted at 2026-09-25 07:24:54 on dist-test-slave-zw07"
server_key: "5c7380fd26345d5ef42e874565ac54bb"
server_key_iv: "aa4be8a11c72d1421e6b372d58f309d4"
server_key_version: "encryptionkey@0"
I20260925 07:24:54.870639 25819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:24:54.871536 25834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:54.871752 25819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:24:54.871855 25819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "73259d9904864c309923ad9dc434d19c"
format_stamp: "Formatted at 2026-09-25 07:24:54 on dist-test-slave-zw07"
server_key: "5c7380fd26345d5ef42e874565ac54bb"
server_key_iv: "aa4be8a11c72d1421e6b372d58f309d4"
server_key_version: "encryptionkey@0"
I20260925 07:24:54.871917 25819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:54.900756 25819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:54.903609 25819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:54.903752 25819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:54.908136 25819 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:35935
I20260925 07:24:54.908172 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:35935 every 8 connection(s)
I20260925 07:24:54.908579 25819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:54.909546 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:54.912037 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: Bootstrap starting.
I20260925 07:24:54.912381 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25819
I20260925 07:24:54.912524 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:54.912760 4284 external_mini_cluster.cc:1428] Setting key 7659aad70c1e7774de04ad6f4f867e91
I20260925 07:24:54.912779 25887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:54.913069 25887 log.cc:824] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:54.913884 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: No bootstrap required, opened a new log
I20260925 07:24:54.915391 25887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } }
I20260925 07:24:54.915562 25887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:54.915613 25887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73259d9904864c309923ad9dc434d19c, State: Initialized, Role: FOLLOWER
I20260925 07:24:54.915805 25887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } }
I20260925 07:24:54.915912 25887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:54.915958 25887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:54.916010 25887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:54.916569 25887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } }
I20260925 07:24:54.916692 25887 leader_election.cc:302] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73259d9904864c309923ad9dc434d19c; no voters:
I20260925 07:24:54.916911 25887 leader_election.cc:288] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:54.916942 25892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:54.917063 25892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [term 1 LEADER]: Becoming Leader. State: Replica: 73259d9904864c309923ad9dc434d19c, State: Running, Role: LEADER
I20260925 07:24:54.917145 25892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } }
I20260925 07:24:54.917184 25887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:54.917511 25892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73259d9904864c309923ad9dc434d19c. Latest consensus state: current_term: 1 leader_uuid: "73259d9904864c309923ad9dc434d19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } } }
I20260925 07:24:54.917601 25892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:54.917523 25894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73259d9904864c309923ad9dc434d19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73259d9904864c309923ad9dc434d19c" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35935 } } }
I20260925 07:24:54.917761 25894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:54.919032 25907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:54.919162 25907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:54.919361 25898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:54.919567 25898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:24:54.921106 25898 catalog_manager.cc:1347] Generated new cluster ID: 7ef1785f751d436e9cdfb3804f0bce6b
I20260925 07:24:54.921151 25898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:54.926086 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:54.913346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44073 (local address 127.4.47.62:35935)
0925 07:24:54.913535 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:54.913547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:54.913588 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:24:54.914149 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:54.914156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:54.914213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:54.914381 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:54.915104 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:54.916600 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:54.917794 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:54.918304 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:54.918372 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:54.921706 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:54.921710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:54.921710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:54.921724 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:54.924204 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:54.924668 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:54.924671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:54.924672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:54.924797 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:54.925044 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:54.925045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:54.925045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:54.925408 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:54.925583 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:54.925716 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 07:24:54.925836 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260925 07:24:54.927578 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:54.928181 25898 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:54.928952 25898 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:54.936053 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:54.929243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42291 (local address 127.4.47.62:35935)
0925 07:24:54.929282 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:54.929284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:54.929433 (+ 149us) server_negotiation.cc:407] Connection header received
0925 07:24:54.929477 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:54.929478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:54.929488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:54.929566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:54.930112 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:54.930618 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:54.931471 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:54.931694 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:54.931797 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:54.933290 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:54.933291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:54.933291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:54.933299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:54.934766 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:54.935187 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:54.935189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:54.935189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:54.935262 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:54.935500 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:54.935501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:54.935501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:54.935615 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:54.935733 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:54.935863 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 07:24:54.935930 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:24:54.936960 25898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73259d9904864c309923ad9dc434d19c: Generated new TSK 0
I20260925 07:24:54.937160 25898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:54.942101 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25819
W20260925 07:24:54.946883 25804 connection.cc:535] client connection to 127.4.47.62:35935 recv error: Network error: failed to read from TLS socket (remote: 127.4.47.62:35935): Connection reset by peer (error 104)
2026-09-25T07:24:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25918](info): set up 2 sockets
Sep 25 07:24:54 dist-test-slave-zw07 krb5kdc[25918](info): commencing operation
krb5kdc: starting...
W20260925 07:24:55.979223 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:56 dist-test-slave-zw07 krb5kdc[25918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:56.102247 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38729
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:56.184840 25931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:56.185097 25931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:56.185137 25931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:56.186329 25931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:56.186403 25931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:56.186429 25931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:56.186456 25931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:56.186487 25931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:56.188024 25931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:56.188315 25931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:56.188552 25931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:24:56.193648 25931 server_base.cc:1034] running on GCE node
W20260925 07:24:56.193608 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:56.193619 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:56.193604 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:56.194129 25931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:56.194396 25931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:56.195533 25931 hybrid_clock.cc:648] HybridClock initialized: now 1790321096195514 us; error 40 us; skew 500 ppm
Sep 25 07:24:56 dist-test-slave-zw07 krb5kdc[25918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:56.198346 25931 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:56.198890 25931 webserver.cc:460] Webserver started at http://127.4.47.62:34141/ using document root <none> and password file <none>
I20260925 07:24:56.199184 25931 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:56.199234 25931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:56.199388 25931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:56.200508 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "2ff1456933cb4a2ea66eb2a36a69cf43"
format_stamp: "Formatted at 2026-09-25 07:24:56 on dist-test-slave-zw07"
server_key: "0c4939932cbecfcb8837530147e0e0b8"
server_key_iv: "044880bae590b435b3fff2a64203f32b"
server_key_version: "encryptionkey@0"
I20260925 07:24:56.200887 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "2ff1456933cb4a2ea66eb2a36a69cf43"
format_stamp: "Formatted at 2026-09-25 07:24:56 on dist-test-slave-zw07"
server_key: "0c4939932cbecfcb8837530147e0e0b8"
server_key_iv: "044880bae590b435b3fff2a64203f32b"
server_key_version: "encryptionkey@0"
I20260925 07:24:56.202328 25931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:24:56.203207 25946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:56.203392 25931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:24:56.203495 25931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "2ff1456933cb4a2ea66eb2a36a69cf43"
format_stamp: "Formatted at 2026-09-25 07:24:56 on dist-test-slave-zw07"
server_key: "0c4939932cbecfcb8837530147e0e0b8"
server_key_iv: "044880bae590b435b3fff2a64203f32b"
server_key_version: "encryptionkey@0"
I20260925 07:24:56.203581 25931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:56.213726 25931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:56.216203 25931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:56.216378 25931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:56.220722 25931 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:38729
I20260925 07:24:56.220758 25998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:38729 every 8 connection(s)
I20260925 07:24:56.221120 25931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:56.222057 25999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:56.224475 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: Bootstrap starting.
I20260925 07:24:56.225021 25999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:56.225255 25999 log.cc:824] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:56.225958 25999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: No bootstrap required, opened a new log
I20260925 07:24:56.227339 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 25931
I20260925 07:24:56.227481 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:56.227662 4284 external_mini_cluster.cc:1428] Setting key 266313b90694e5e1a21d792b6dcaca92
I20260925 07:24:56.227736 25999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } }
I20260925 07:24:56.227928 25999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:56.227969 25999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff1456933cb4a2ea66eb2a36a69cf43, State: Initialized, Role: FOLLOWER
I20260925 07:24:56.228188 25999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } }
I20260925 07:24:56.228293 25999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:56.228343 25999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:56.228427 25999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:56.229367 25999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } }
I20260925 07:24:56.229494 25999 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff1456933cb4a2ea66eb2a36a69cf43; no voters:
I20260925 07:24:56.229689 25999 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:56.229732 26004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:56.229840 26004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [term 1 LEADER]: Becoming Leader. State: Replica: 2ff1456933cb4a2ea66eb2a36a69cf43, State: Running, Role: LEADER
I20260925 07:24:56.229918 26004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } }
I20260925 07:24:56.229955 25999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:56.230276 26004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } } }
I20260925 07:24:56.230350 26004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:56.230468 26004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff1456933cb4a2ea66eb2a36a69cf43. Latest consensus state: current_term: 1 leader_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff1456933cb4a2ea66eb2a36a69cf43" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38729 } } }
I20260925 07:24:56.230538 26004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43 [sys.catalog]: This master's current role is: LEADER
W20260925 07:24:56.231842 26020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:56.231920 26020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:56.231982 26009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:56.232161 26009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:56.233845 26009 catalog_manager.cc:1347] Generated new cluster ID: 5d219f575d4849dbb0b9c9ee16abd1c8
I20260925 07:24:56.233906 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:56.228207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54389 (local address 127.4.47.62:38729)
0925 07:24:56.228424 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:56.228435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:56.228578 (+ 143us) server_negotiation.cc:407] Connection header received
0925 07:24:56.228728 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:56.228740 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:56.228794 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:56.228928 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:56.229126 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:56.230425 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:56.231607 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:56.232217 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:56.232667 (+ 450us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:56.233154 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:56.233158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:56.233159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:56.233367 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:56.233404 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:56.233419 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:56.233508 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:56.233652 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 07:24:56.233681 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":129,"threads_started":1}
I20260925 07:24:56.233912 26009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:56.234836 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:56.246290 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:56.236296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32775 (local address 127.4.47.62:38729)
0925 07:24:56.243203 (+ 6907us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:56.243205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:56.243219 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:24:56.243245 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:56.243246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:56.243256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:56.243368 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:56.243547 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:56.244042 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:56.244909 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:56.245143 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:56.245232 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:56.246019 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:56.246021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:56.246021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:56.246026 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:56.246042 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:56.246047 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:56.246100 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:56.246160 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:24:56.246173 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6868}
I20260925 07:24:56.255128 26009 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:56.255712 26009 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:56.261055 26009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff1456933cb4a2ea66eb2a36a69cf43: Generated new TSK 0
I20260925 07:24:56.261305 26009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:56.265420 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 25931
W20260925 07:24:56.269709 25916 connection.cc:535] client connection to 127.4.47.62:38729 recv error: Network error: failed to read from TLS socket (remote: 127.4.47.62:38729): Connection reset by peer (error 104)
2026-09-25T07:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:56 dist-test-slave-zw07 krb5kdc[26030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:56 dist-test-slave-zw07 krb5kdc[26030](info): set up 2 sockets
Sep 25 07:24:56 dist-test-slave-zw07 krb5kdc[26030](info): commencing operation
krb5kdc: starting...
W20260925 07:24:57.303254 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:57 dist-test-slave-zw07 krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:57.426210 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:43261
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:57.509160 26043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:57.509387 26043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:57.509413 26043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:57.510541 26043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:57.510578 26043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:57.510592 26043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:57.510605 26043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:57.510617 26043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:57.512131 26043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:43261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:57.512351 26043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:57.512528 26043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:57.517551 26049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:57.517619 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:57.517570 26048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:57.517774 26043 server_base.cc:1034] running on GCE node
I20260925 07:24:57.518040 26043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:57.518338 26043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:57.519515 26043 hybrid_clock.cc:648] HybridClock initialized: now 1790321097519494 us; error 46 us; skew 500 ppm
Sep 25 07:24:57 dist-test-slave-zw07 krb5kdc[26030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:57.522047 26043 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:57.522529 26043 webserver.cc:460] Webserver started at http://127.4.47.62:33875/ using document root <none> and password file <none>
I20260925 07:24:57.522769 26043 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:57.522840 26043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:57.522993 26043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:57.524099 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "1ec7b78054464cd7beb7ff29ce96f500"
format_stamp: "Formatted at 2026-09-25 07:24:57 on dist-test-slave-zw07"
server_key: "f8f190c2333582eb64881467379373e2"
server_key_iv: "67bfe0ba77621a3b652acf155b0452af"
server_key_version: "encryptionkey@0"
I20260925 07:24:57.524478 26043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "1ec7b78054464cd7beb7ff29ce96f500"
format_stamp: "Formatted at 2026-09-25 07:24:57 on dist-test-slave-zw07"
server_key: "f8f190c2333582eb64881467379373e2"
server_key_iv: "67bfe0ba77621a3b652acf155b0452af"
server_key_version: "encryptionkey@0"
I20260925 07:24:57.525841 26043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:57.526638 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:57.526832 26043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:57.526933 26043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "1ec7b78054464cd7beb7ff29ce96f500"
format_stamp: "Formatted at 2026-09-25 07:24:57 on dist-test-slave-zw07"
server_key: "f8f190c2333582eb64881467379373e2"
server_key_iv: "67bfe0ba77621a3b652acf155b0452af"
server_key_version: "encryptionkey@0"
I20260925 07:24:57.527016 26043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:57.530411 26043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:57.532773 26043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:57.532950 26043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:57.537011 26043 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:43261
I20260925 07:24:57.537042 26110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:43261 every 8 connection(s)
I20260925 07:24:57.537379 26043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:57.538341 26111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:57.540733 26111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500: Bootstrap starting.
I20260925 07:24:57.541029 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26043
I20260925 07:24:57.541172 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:57.541337 4284 external_mini_cluster.cc:1428] Setting key d2dbbae8191fa8c14ea23e4d1db959c8
I20260925 07:24:57.541491 26111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:57.541836 26111 log.cc:824] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:57.542636 26111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500: No bootstrap required, opened a new log
I20260925 07:24:57.543978 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:57.541939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58701 (local address 127.4.47.62:43261)
0925 07:24:57.542150 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:57.542160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:57.542196 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:24:57.542670 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:57.542677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:57.542703 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:57.542863 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:57.543506 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:57.543509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:57.543510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:57.543513 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:57.543549 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:57.543558 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:57.543624 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:57.543729 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 07:24:57.543801 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 07:24:57.545066 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:57.545343 26111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } }
I20260925 07:24:57.545495 26111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:57.545540 26111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec7b78054464cd7beb7ff29ce96f500, State: Initialized, Role: FOLLOWER
I20260925 07:24:57.545661 26111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } }
I20260925 07:24:57.545751 26111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:57.545795 26111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:57.545845 26111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:57.546463 26111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } }
I20260925 07:24:57.546582 26111 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec7b78054464cd7beb7ff29ce96f500; no voters:
I20260925 07:24:57.546751 26111 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:57.546800 26121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:57.546952 26121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec7b78054464cd7beb7ff29ce96f500, State: Running, Role: LEADER
I20260925 07:24:57.546991 26111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:57.547055 26121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } }
I20260925 07:24:57.547559 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec7b78054464cd7beb7ff29ce96f500. Latest consensus state: current_term: 1 leader_uuid: "1ec7b78054464cd7beb7ff29ce96f500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } } }
I20260925 07:24:57.547669 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:57.547767 26122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec7b78054464cd7beb7ff29ce96f500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec7b78054464cd7beb7ff29ce96f500" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 43261 } } }
I20260925 07:24:57.547897 26130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:57.547902 26122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:57.548084 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:57.546591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33603 (local address 127.4.47.62:43261)
0925 07:24:57.546687 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:57.546689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:57.546747 (+ 58us) server_negotiation.cc:407] Connection header received
0925 07:24:57.546838 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:57.546839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:57.546845 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:57.546892 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:57.547780 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:57.547781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:57.547781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:24:57.547786 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:57.547801 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:24:57.547806 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:57.547855 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:57.547997 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 07:24:57.548010 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:24:57.548573 26130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:57.550292 26130 catalog_manager.cc:1347] Generated new cluster ID: 8ba8b82d130b4cff8e070a5781a20e93
I20260925 07:24:57.550354 26130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:57.561007 26130 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:57.561206 26130 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:57.567701 26130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec7b78054464cd7beb7ff29ce96f500: Generated new TSK 0
I20260925 07:24:57.567929 26130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:57.575577 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26043
2026-09-25T07:24:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:57 dist-test-slave-zw07 krb5kdc[26142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:57 dist-test-slave-zw07 krb5kdc[26142](info): set up 2 sockets
Sep 25 07:24:57 dist-test-slave-zw07 krb5kdc[26142](info): commencing operation
krb5kdc: starting...
W20260925 07:24:58.607329 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:24:58.730032 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37835
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:24:58.819120 26155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:24:58.819402 26155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:24:58.819442 26155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:24:58.820703 26155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:24:58.820786 26155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:24:58.820813 26155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:24:58.820847 26155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:24:58.820869 26155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:24:58.822510 26155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:24:58.822826 26155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:24:58.823117 26155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:24:58.828321 26161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:58.828364 26155 server_base.cc:1034] running on GCE node
W20260925 07:24:58.828329 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:24:58.828329 26163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:24:58.828752 26155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:24:58.829023 26155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:24:58.830186 26155 hybrid_clock.cc:648] HybridClock initialized: now 1790321098830171 us; error 28 us; skew 500 ppm
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:24:58.832975 26155 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:24:58.833492 26155 webserver.cc:460] Webserver started at http://127.4.47.62:43653/ using document root <none> and password file <none>
I20260925 07:24:58.833742 26155 fs_manager.cc:359] Metadata directory not provided
I20260925 07:24:58.833814 26155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:24:58.833966 26155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:24:58.835227 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "59fea56e53214e86a05544ce78d798a9"
format_stamp: "Formatted at 2026-09-25 07:24:58 on dist-test-slave-zw07"
server_key: "dfd563742ef5e09c9c09981e20b04244"
server_key_iv: "c7dff5b667838012050555e9ede0e5bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:58.835603 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "59fea56e53214e86a05544ce78d798a9"
format_stamp: "Formatted at 2026-09-25 07:24:58 on dist-test-slave-zw07"
server_key: "dfd563742ef5e09c9c09981e20b04244"
server_key_iv: "c7dff5b667838012050555e9ede0e5bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:58.837165 26155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:24:58.838076 26170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:24:58.838260 26155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:24:58.838375 26155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "59fea56e53214e86a05544ce78d798a9"
format_stamp: "Formatted at 2026-09-25 07:24:58 on dist-test-slave-zw07"
server_key: "dfd563742ef5e09c9c09981e20b04244"
server_key_iv: "c7dff5b667838012050555e9ede0e5bb"
server_key_version: "encryptionkey@0"
I20260925 07:24:58.838457 26155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:24:58.854522 26155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:24:58.856839 26155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:24:58.857021 26155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:24:58.861258 26155 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:37835
I20260925 07:24:58.861294 26222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:37835 every 8 connection(s)
I20260925 07:24:58.861683 26155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:24:58.863196 26223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:24:58.865746 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26155
I20260925 07:24:58.865895 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:24:58.866104 4284 external_mini_cluster.cc:1428] Setting key f5ff495e04dfcab6b623b2340a9a686e
I20260925 07:24:58.866101 26223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: Bootstrap starting.
I20260925 07:24:58.866863 26223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: Neither blocks nor log segments found. Creating new log.
I20260925 07:24:58.867213 26223 log.cc:824] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:24:58.867987 26223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: No bootstrap required, opened a new log
I20260925 07:24:58.870357 26223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } }
I20260925 07:24:58.870496 26223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:24:58.870518 26223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fea56e53214e86a05544ce78d798a9, State: Initialized, Role: FOLLOWER
I20260925 07:24:58.870664 26223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } }
I20260925 07:24:58.870738 26223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:24:58.870760 26223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:24:58.870814 26223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:24:58.871454 26223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } }
I20260925 07:24:58.871578 26223 leader_election.cc:302] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fea56e53214e86a05544ce78d798a9; no voters:
I20260925 07:24:58.871794 26223 leader_election.cc:288] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:24:58.871814 26228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:24:58.871982 26228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [term 1 LEADER]: Becoming Leader. State: Replica: 59fea56e53214e86a05544ce78d798a9, State: Running, Role: LEADER
I20260925 07:24:58.872125 26223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:24:58.872123 26228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } }
I20260925 07:24:58.873425 26230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59fea56e53214e86a05544ce78d798a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } } }
I20260925 07:24:58.873530 26230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:24:58.873915 26229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fea56e53214e86a05544ce78d798a9. Latest consensus state: current_term: 1 leader_uuid: "59fea56e53214e86a05544ce78d798a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fea56e53214e86a05544ce78d798a9" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37835 } } }
I20260925 07:24:58.873987 26229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9 [sys.catalog]: This master's current role is: LEADER
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
W20260925 07:24:58.874119 26243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:24:58.874205 26243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:24:58.874287 26244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:24:58.874492 26244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:24:58.876176 26244 catalog_manager.cc:1347] Generated new cluster ID: d5074e850b194fa98d0e90af995186df
I20260925 07:24:58.876233 26244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:24:58.878260 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:58.866698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38143 (local address 127.4.47.62:37835)
0925 07:24:58.866940 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:58.866950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:58.866991 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:24:58.867611 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:58.867619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:58.867713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:58.867869 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:58.868525 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:58.869986 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:58.870919 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:58.871434 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:58.871576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:58.874651 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:58.874656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:58.874656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:58.874669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:58.876646 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:58.877028 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:58.877031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:58.877031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:58.877134 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:58.877351 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:58.877352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:58.877353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:58.877605 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:58.877771 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:58.877882 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 07:24:58.878031 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":111,"threads_started":1}
I20260925 07:24:58.880316 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:24:58.890040 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:24:58.881763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34969 (local address 127.4.47.62:37835)
0925 07:24:58.881802 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:24:58.881804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:24:58.883320 (+ 1516us) server_negotiation.cc:407] Connection header received
0925 07:24:58.883398 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:24:58.883399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:24:58.883409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:24:58.883466 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:24:58.884038 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:58.884629 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:58.885372 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:24:58.885607 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:24:58.885687 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:24:58.887293 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:24:58.887295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:24:58.887295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:24:58.887303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:24:58.888816 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:58.889190 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:58.889192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:58.889192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:58.889250 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:24:58.889458 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:24:58.889459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:24:58.889459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:24:58.889576 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:24:58.889744 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:24:58.889843 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 07:24:58.889928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:24:58.898833 26244 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:24:58.899650 26244 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:24:58.902820 26244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59fea56e53214e86a05544ce78d798a9: Generated new TSK 0
I20260925 07:24:58.903024 26244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:24:58.909118 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26155
2026-09-25T07:24:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26254](info): set up 2 sockets
Sep 25 07:24:58 dist-test-slave-zw07 krb5kdc[26254](info): commencing operation
krb5kdc: starting...
W20260925 07:24:59.947247 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:25:00 dist-test-slave-zw07 krb5kdc[26254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:25:00.074671 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:35613
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:25:00.157807 26267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:00.158082 26267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:00.158123 26267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:00.159392 26267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:00.159457 26267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:00.159482 26267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:00.159504 26267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:00.159567 26267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:00.161167 26267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:00.161459 26267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:00.161701 26267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:00.166832 26272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:00.166848 26275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:00.166832 26273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:00.167135 26267 server_base.cc:1034] running on GCE node
I20260925 07:25:00.167387 26267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:00.167706 26267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:00.168884 26267 hybrid_clock.cc:648] HybridClock initialized: now 1790321100168867 us; error 48 us; skew 500 ppm
Sep 25 07:25:00 dist-test-slave-zw07 krb5kdc[26254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321100, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:00.179970 26267 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:25:00.180560 26267 webserver.cc:460] Webserver started at http://127.4.47.62:37217/ using document root <none> and password file <none>
I20260925 07:25:00.180802 26267 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:00.180876 26267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:00.181039 26267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:00.182150 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "f65c710160b1488ba21352571c7df836"
format_stamp: "Formatted at 2026-09-25 07:25:00 on dist-test-slave-zw07"
server_key: "8bd4d60851fdbcfe76689f4c20d2bf2f"
server_key_iv: "a566b420571f98302f5242ef24a02942"
server_key_version: "encryptionkey@0"
I20260925 07:25:00.182523 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "f65c710160b1488ba21352571c7df836"
format_stamp: "Formatted at 2026-09-25 07:25:00 on dist-test-slave-zw07"
server_key: "8bd4d60851fdbcfe76689f4c20d2bf2f"
server_key_iv: "a566b420571f98302f5242ef24a02942"
server_key_version: "encryptionkey@0"
I20260925 07:25:00.184026 26267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:25:00.184846 26282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:00.185019 26267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:00.185135 26267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "f65c710160b1488ba21352571c7df836"
format_stamp: "Formatted at 2026-09-25 07:25:00 on dist-test-slave-zw07"
server_key: "8bd4d60851fdbcfe76689f4c20d2bf2f"
server_key_iv: "a566b420571f98302f5242ef24a02942"
server_key_version: "encryptionkey@0"
I20260925 07:25:00.185213 26267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:00.198031 26267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:00.200892 26267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:00.201097 26267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:00.205777 26267 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:35613
I20260925 07:25:00.205862 26334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:35613 every 8 connection(s)
I20260925 07:25:00.206260 26267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:25:00.207247 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:00.209965 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26267
I20260925 07:25:00.210029 26335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836: Bootstrap starting.
I20260925 07:25:00.210094 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:25:00.210279 4284 external_mini_cluster.cc:1428] Setting key a1fefc227bd796d45c42b5660af89505
I20260925 07:25:00.210741 26335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:00.211278 26335 log.cc:824] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:00.211966 26335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836: No bootstrap required, opened a new log
I20260925 07:25:00.213798 26335 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } }
I20260925 07:25:00.213963 26335 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:25:00.214015 26335 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65c710160b1488ba21352571c7df836, State: Initialized, Role: FOLLOWER
I20260925 07:25:00.214181 26335 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } }
I20260925 07:25:00.214268 26335 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:25:00.214298 26335 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:25:00.214332 26335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:00.215058 26335 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } }
I20260925 07:25:00.215216 26335 leader_election.cc:302] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65c710160b1488ba21352571c7df836; no voters:
I20260925 07:25:00.215392 26335 leader_election.cc:288] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:25:00.215445 26340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:25:00.215663 26340 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [term 1 LEADER]: Becoming Leader. State: Replica: f65c710160b1488ba21352571c7df836, State: Running, Role: LEADER
I20260925 07:25:00.215792 26340 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } }
I20260925 07:25:00.215672 26335 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:00.216195 26341 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65c710160b1488ba21352571c7df836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } } }
I20260925 07:25:00.216208 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65c710160b1488ba21352571c7df836. Latest consensus state: current_term: 1 leader_uuid: "f65c710160b1488ba21352571c7df836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c710160b1488ba21352571c7df836" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 35613 } } }
I20260925 07:25:00.216289 26341 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:00.216300 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:00.216701 26345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:25:00.216706 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:00.210888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44161 (local address 127.4.47.62:35613)
0925 07:25:00.211149 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:00.211160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:00.211202 (+ 42us) server_negotiation.cc:407] Connection header received
0925 07:25:00.211737 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:00.211745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:00.211807 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:00.211970 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:00.212238 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:00.213212 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:00.214126 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:00.214765 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:00.214907 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:00.215712 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:00.215716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:00.215717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:25:00.215950 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:00.215992 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:25:00.216009 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:00.216058 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:00.216238 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 07:25:00.216320 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260925 07:25:00.217680 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:25:00.219276 26345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:25:00.221105 26345 catalog_manager.cc:1347] Generated new cluster ID: eaeebeeadddb40bcbade6e65fe15f6e8
I20260925 07:25:00.221161 26345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:25:00.223393 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:00.219316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58863 (local address 127.4.47.62:35613)
0925 07:25:00.219358 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:00.219361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:00.220193 (+ 832us) server_negotiation.cc:407] Connection header received
0925 07:25:00.220243 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:00.220244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:00.220256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:00.220306 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:00.220460 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:00.221236 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:00.222011 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:00.222249 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:00.222342 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:00.223117 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:00.223118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:00.223119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:25:00.223125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:00.223141 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:25:00.223146 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:00.223207 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:00.223272 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 07:25:00.223296 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:25:00.237852 26345 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:25:00.238505 26345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:25:00.242029 26345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65c710160b1488ba21352571c7df836: Generated new TSK 0
I20260925 07:25:00.242242 26345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:25:00.246111 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26267
2026-09-25T07:25:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:25:00 dist-test-slave-zw07 krb5kdc[26366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:25:00 dist-test-slave-zw07 krb5kdc[26366](info): set up 2 sockets
Sep 25 07:25:00 dist-test-slave-zw07 krb5kdc[26366](info): commencing operation
krb5kdc: starting...
W20260925 07:25:01.311307 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:25:01 dist-test-slave-zw07 krb5kdc[26366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:25:01.434986 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:46789
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:25:01.517125 26379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:01.517354 26379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:01.517379 26379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:01.518615 26379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:01.518667 26379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:01.518682 26379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:01.518695 26379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:01.518708 26379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:01.520450 26379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:46789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:01.520718 26379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:01.520957 26379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:01.526176 26387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:01.526141 26384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:01.526193 26379 server_base.cc:1034] running on GCE node
W20260925 07:25:01.526162 26385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:01.526685 26379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:01.526968 26379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:01.528133 26379 hybrid_clock.cc:648] HybridClock initialized: now 1790321101528119 us; error 28 us; skew 500 ppm
Sep 25 07:25:01 dist-test-slave-zw07 krb5kdc[26366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:01.530671 26379 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:25:01.531236 26379 webserver.cc:460] Webserver started at http://127.4.47.62:33945/ using document root <none> and password file <none>
I20260925 07:25:01.531473 26379 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:01.531518 26379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:01.531667 26379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:01.532780 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "81187ebeecd54c3c83ee9b9b281ea9a8"
format_stamp: "Formatted at 2026-09-25 07:25:01 on dist-test-slave-zw07"
server_key: "7bdd1331953669cbdcd0bc82151d3a3b"
server_key_iv: "bffce84fd3dce218569b08498ff5899e"
server_key_version: "encryptionkey@0"
I20260925 07:25:01.533154 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "81187ebeecd54c3c83ee9b9b281ea9a8"
format_stamp: "Formatted at 2026-09-25 07:25:01 on dist-test-slave-zw07"
server_key: "7bdd1331953669cbdcd0bc82151d3a3b"
server_key_iv: "bffce84fd3dce218569b08498ff5899e"
server_key_version: "encryptionkey@0"
I20260925 07:25:01.534538 26379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:25:01.535387 26394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:01.535614 26379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:01.535722 26379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "81187ebeecd54c3c83ee9b9b281ea9a8"
format_stamp: "Formatted at 2026-09-25 07:25:01 on dist-test-slave-zw07"
server_key: "7bdd1331953669cbdcd0bc82151d3a3b"
server_key_iv: "bffce84fd3dce218569b08498ff5899e"
server_key_version: "encryptionkey@0"
I20260925 07:25:01.535809 26379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:01.539306 26379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:01.541416 26379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:01.541603 26379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:01.545560 26379 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:46789
I20260925 07:25:01.545593 26446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:46789 every 8 connection(s)
I20260925 07:25:01.545967 26379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:25:01.546897 26447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:01.549240 26447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: Bootstrap starting.
I20260925 07:25:01.549824 26447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:01.550096 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26379
I20260925 07:25:01.550168 26447 log.cc:824] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:01.550222 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:25:01.550418 4284 external_mini_cluster.cc:1428] Setting key 51f7391bbf1c43e1f6fa96a83f371011
I20260925 07:25:01.551153 26447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: No bootstrap required, opened a new log
I20260925 07:25:01.553098 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:01.550999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47901 (local address 127.4.47.62:46789)
0925 07:25:01.551221 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:01.551231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:01.551272 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:25:01.551757 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:01.551764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:01.551792 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:01.551944 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:01.552614 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:01.552619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:01.552619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:25:01.552623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:01.552670 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:25:01.552680 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:01.552750 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:01.552827 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 07:25:01.552882 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":126,"threads_started":1}
I20260925 07:25:01.554162 26447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } }
I20260925 07:25:01.554356 26447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:25:01.554325 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:25:01.554385 26447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81187ebeecd54c3c83ee9b9b281ea9a8, State: Initialized, Role: FOLLOWER
I20260925 07:25:01.554526 26447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } }
I20260925 07:25:01.554612 26447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:25:01.554636 26447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:25:01.554689 26447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:01.555341 26447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } }
I20260925 07:25:01.555430 26447 leader_election.cc:302] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81187ebeecd54c3c83ee9b9b281ea9a8; no voters:
I20260925 07:25:01.555624 26447 leader_election.cc:288] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:25:01.555684 26454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:25:01.555841 26454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 81187ebeecd54c3c83ee9b9b281ea9a8, State: Running, Role: LEADER
I20260925 07:25:01.555884 26447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:01.555943 26454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } }
I20260925 07:25:01.556272 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81187ebeecd54c3c83ee9b9b281ea9a8. Latest consensus state: current_term: 1 leader_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } } }
I20260925 07:25:01.556358 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:01.556282 26458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81187ebeecd54c3c83ee9b9b281ea9a8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 46789 } } }
I20260925 07:25:01.556403 26458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8 [sys.catalog]: This master's current role is: LEADER
W20260925 07:25:01.557690 26473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:01.557750 26473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:01.557819 26464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:25:01.557893 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:01.555966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41773 (local address 127.4.47.62:46789)
0925 07:25:01.556001 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:01.556003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:01.556650 (+ 647us) server_negotiation.cc:407] Connection header received
0925 07:25:01.556741 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:01.556743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:01.556749 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:01.556815 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:01.557548 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:01.557550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:01.557550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:25:01.557555 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:01.557571 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:25:01.557576 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:01.557611 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:01.557712 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 07:25:01.557728 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:25:01.557999 26464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:25:01.559590 26464 catalog_manager.cc:1347] Generated new cluster ID: fb99dd02352a4a62999fe979b459b777
I20260925 07:25:01.559681 26464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:25:01.572230 26464 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:25:01.572443 26464 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:25:01.574656 26464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81187ebeecd54c3c83ee9b9b281ea9a8: Generated new TSK 0
I20260925 07:25:01.574859 26464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:25:01.580840 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26379
2026-09-25T07:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1334 ms)
[----------] 18 tests from AuthTokenIssuingTest (24048 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:25:01 dist-test-slave-zw07 krb5kdc[26478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:25:01 dist-test-slave-zw07 krb5kdc[26478](info): set up 2 sockets
Sep 25 07:25:01 dist-test-slave-zw07 krb5kdc[26478](info): commencing operation
krb5kdc: starting...
W20260925 07:25:02.607362 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:25:02 dist-test-slave-zw07 krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:25:02.732198 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38939
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:25:02.814957 26491 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:02.815213 26491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:02.815231 26491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:02.815253 26491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:02.816380 26491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:02.816417 26491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:02.816431 26491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:02.816444 26491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:02.816457 26491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:02.818017 26491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:38939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:02.818225 26491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:02.818405 26491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:02.823616 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:02.823630 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:02.823722 26491 server_base.cc:1034] running on GCE node
W20260925 07:25:02.823632 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:02.824209 26491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:02.824476 26491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:02.825659 26491 hybrid_clock.cc:648] HybridClock initialized: now 1790321102825613 us; error 42 us; skew 500 ppm
Sep 25 07:25:02 dist-test-slave-zw07 krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:02.828447 26491 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:25:02.828958 26491 webserver.cc:460] Webserver started at http://127.4.47.62:34499/ using document root <none> and password file <none>
I20260925 07:25:02.829216 26491 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:02.829262 26491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:02.829433 26491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:02.830539 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "8b726a243c394a1c8d40026c112a0be8"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "447cc0142cb9be0c2b1ee1a895402257"
server_key_iv: "aaddc530d6bf29ad59fbda7f4da5e1b8"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.830921 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "8b726a243c394a1c8d40026c112a0be8"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "447cc0142cb9be0c2b1ee1a895402257"
server_key_iv: "aaddc530d6bf29ad59fbda7f4da5e1b8"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.832337 26491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:25:02.833135 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:02.833317 26491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:02.833405 26491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "8b726a243c394a1c8d40026c112a0be8"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "447cc0142cb9be0c2b1ee1a895402257"
server_key_iv: "aaddc530d6bf29ad59fbda7f4da5e1b8"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.833482 26491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:02.840718 26491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:02.843142 26491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:02.843308 26491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:02.847285 26491 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:38939
I20260925 07:25:02.847396 26558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:38939 every 8 connection(s)
I20260925 07:25:02.847740 26491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:25:02.848775 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:02.850387 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:02 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:02.857337 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26491
I20260925 07:25:02.857421 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:25:02.857635 4284 external_mini_cluster.cc:1428] Setting key 6e56ea3e069394260134cb82bf6a087d
I20260925 07:25:02.860797 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.851179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38939 (local address 127.0.0.1:53878)
0925 07:25:02.851369 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.851413 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:02.851878 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:02.852187 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:02.852189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:02.852438 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:02.852595 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:02.852599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:02.853629 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:02.853630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:02.854319 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:02.854321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:02.854820 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:02.854820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:02.854873 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:02.855444 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:02.855452 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:02.856995 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:02.859124 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:02.859135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:02.859136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:02.859328 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:02.859637 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:02.859638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:02.859638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:02.859716 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:02.860126 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:02.860130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:02.860205 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 07:25:02.860300 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:25:02.860530 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":89,"thread_start_us":86,"threads_started":1}
I20260925 07:25:02.860805 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.851240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.4.47.62:38939)
0925 07:25:02.851442 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:02.851451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:02.851511 (+ 60us) server_negotiation.cc:407] Connection header received
0925 07:25:02.852047 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:02.852050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:02.852067 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:02.852146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:02.852717 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:02.853542 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:02.854404 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:02.854726 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:02.854811 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:02.857076 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:02.857083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:02.857083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:02.857089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:02.858978 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:02.859410 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:02.859414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:02.859415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:02.859543 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:02.859791 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:02.859792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:02.859793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:02.860064 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:02.860132 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:02.860339 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 07:25:02.860471 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260925 07:25:02.861660 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:02.862006 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.861883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54180)
0925 07:25:02.861932 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.861976 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:25:02.862128 26510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
I20260925 07:25:02.862524 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.862434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54192)
0925 07:25:02.862479 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.862503 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:25:02.862625 26559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34609: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.47.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
I20260925 07:25:02.883399 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:34609
--webserver_interface=127.4.47.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:25:02.911859 26559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } attempt: 1
I20260925 07:25:02.912380 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.912240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54200)
0925 07:25:02.912304 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.912349 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 07:25:02.912796 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.912691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54208)
0925 07:25:02.912743 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.912768 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 07:25:02.912881 26559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34609: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
W20260925 07:25:02.964263 26566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:02.964499 26566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:02.964517 26566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:02.964540 26566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:02.966080 26566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:02.966142 26566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:02.966158 26566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:02.966171 26566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:02.966238 26566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:02.967947 26566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.47.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:02.968225 26566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:02.968498 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:02.973500 26572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:02.973562 26574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:02.973572 26571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:02.973759 26566 server_base.cc:1034] running on GCE node
I20260925 07:25:02.974028 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:02.974341 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:02.975497 26566 hybrid_clock.cc:648] HybridClock initialized: now 1790321102975466 us; error 51 us; skew 500 ppm
Sep 25 07:25:02 dist-test-slave-zw07 krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:02.978070 26566 init.cc:377] Logged in from keytab as kudu/127.4.47.61@KRBTEST.COM (short username kudu)
I20260925 07:25:02.978583 26566 webserver.cc:460] Webserver started at http://127.4.47.61:46391/ using document root <none> and password file <none>
I20260925 07:25:02.978822 26566 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:02.978894 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:02.979049 26566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:02.980207 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data/instance:
uuid: "dd0e6d9803e948b1bb7701908d2e1e11"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "b1f8425047b5564861921d4af0d41b14"
server_key_iv: "efc539efd2b7d2deb373907404e31ed2"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.980605 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal/instance:
uuid: "dd0e6d9803e948b1bb7701908d2e1e11"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "b1f8425047b5564861921d4af0d41b14"
server_key_iv: "efc539efd2b7d2deb373907404e31ed2"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.981014 26559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } attempt: 2
I20260925 07:25:02.981436 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.981304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54210)
0925 07:25:02.981363 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.981400 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 07:25:02.981783 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:02.981686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54214)
0925 07:25:02.981733 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:02.981758 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:25:02.981881 26559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:34609: Network error: Client connection negotiation failed: client connection to 127.4.47.61:34609: connect: Connection refused (error 111)
I20260925 07:25:02.982120 26566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:25:02.982929 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:02.983155 26566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:02.983253 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "dd0e6d9803e948b1bb7701908d2e1e11"
format_stamp: "Formatted at 2026-09-25 07:25:02 on dist-test-slave-zw07"
server_key: "b1f8425047b5564861921d4af0d41b14"
server_key_iv: "efc539efd2b7d2deb373907404e31ed2"
server_key_version: "encryptionkey@0"
I20260925 07:25:02.983330 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:02.993023 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:02.995524 26566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:02.995695 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:03.000088 26566 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.61:34609
I20260925 07:25:03.000123 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.61:34609 every 8 connection(s)
I20260925 07:25:03.000509 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
I20260925 07:25:03.001523 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:03.003224 26634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:03.008201 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26566
I20260925 07:25:03.008291 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-1/wal/instance
I20260925 07:25:03.008466 4284 external_mini_cluster.cc:1428] Setting key 9bd2687a6d9f7c624bb83760dafe313e
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:03.013043 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.003985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.4.47.62:38939)
0925 07:25:03.004054 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.004056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.004320 (+ 264us) server_negotiation.cc:407] Connection header received
0925 07:25:03.004822 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.004824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.004837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.004893 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.005550 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.006030 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.007057 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.007342 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.007399 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.009835 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.009837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.009837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.009846 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.011459 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.011992 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.011993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.011994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.012059 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.012327 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.012328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.012329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.012459 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.012533 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.012840 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.012931 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:25:03.013291 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.003970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38939 (local address 127.0.0.1:53886)
0925 07:25:03.004189 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.004230 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.004723 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.004951 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.004954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.005246 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.005458 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.005463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.006128 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.006128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.006971 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.006973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.007434 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.007436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.007554 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.008168 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.008177 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.009724 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.011569 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.011579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.011581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.011912 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.012131 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.012132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.012132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.012259 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.012534 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.012542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.012699 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.012805 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.012984 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260925 07:25:03.013913 26634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.47.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.60@KRBTEST.COM" created.
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:03.022174 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.014255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54220)
0925 07:25:03.014490 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.014501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.014560 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.014731 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.014733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.014888 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.014957 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.014959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.015881 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.015883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.016543 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.016545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.017080 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.017081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.017153 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.017537 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.017541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.019235 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.021163 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.021164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.021164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.021323 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.021552 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.021553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.021553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.021601 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.021935 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.021935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.021971 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.022021 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.022098 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260925 07:25:03.022336 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.014334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.4.47.61:34609)
0925 07:25:03.014505 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.014509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.014554 (+ 45us) server_negotiation.cc:407] Connection header received
0925 07:25:03.014618 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.014620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.014631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.014704 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.015034 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.015791 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.016616 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.017010 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.017070 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.019333 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.019343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.019343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.019350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.021082 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.021415 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.021419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.021419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.021485 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.021660 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.021661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.021661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.021869 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.021956 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.022079 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.022178 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260925 07:25:03.022990 26634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:03.023411 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.023267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57338)
0925 07:25:03.023323 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.023377 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 07:25:03.023578 26584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
I20260925 07:25:03.024084 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.023966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57354)
0925 07:25:03.024019 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.024053 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 07:25:03.024259 26634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:34843: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
I20260925 07:25:03.034732 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:34843
--webserver_interface=127.4.47.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:25:03.073127 26559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } attempt: 3
I20260925 07:25:03.073551 26634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } attempt: 1
I20260925 07:25:03.073935 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.073820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57368)
0925 07:25:03.073861 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.073895 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:25:03.074294 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.074191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57370)
0925 07:25:03.074242 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.074268 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 07:25:03.074394 26634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:34843: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:03.081080 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.073597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.4.47.61:34609)
0925 07:25:03.073724 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.073726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.073734 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:25:03.073762 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.073763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.073774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.073831 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.074266 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.075147 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.075880 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.076203 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.076275 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.078247 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.078248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.078249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.078257 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.079851 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.080249 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.080250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.080250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.080311 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.080522 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.080523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.080524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.080653 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.080727 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.080898 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.080979 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 07:25:03.081081 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.073556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54226)
0925 07:25:03.073622 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.073633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.073682 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.073869 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.073870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.074069 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.074150 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.074152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.075272 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.075273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.075800 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.075802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.076279 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.076280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.076335 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.076923 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.076928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.078162 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.079964 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.079965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.079966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.080158 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.080389 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.080390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.080390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.080458 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.080730 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.080731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.080780 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.080859 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.080975 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 07:25:03.081583 26559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:03.081948 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.081842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57380)
0925 07:25:03.081892 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.081920 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:25:03.082299 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.082211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57384)
0925 07:25:03.082256 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.082278 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:25:03.082396 26559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:34843: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
I20260925 07:25:03.113554 26559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } attempt: 1
I20260925 07:25:03.114096 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.113925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57390)
0925 07:25:03.113997 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.114053 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 07:25:03.114533 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.114411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57400)
0925 07:25:03.114463 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.114507 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:25:03.114668 26559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:34843: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
W20260925 07:25:03.120579 26641 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:03.120803 26641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:03.120821 26641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:03.120875 26641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:03.122048 26641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:03.122092 26641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:03.122105 26641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:03.122153 26641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:03.122181 26641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:03.123766 26641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:38939,127.4.47.61:34609,127.4.47.60:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.47.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:03.124040 26641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:03.124294 26641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:03.129201 26647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:03.129242 26646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:03.129262 26649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:03.129479 26641 server_base.cc:1034] running on GCE node
I20260925 07:25:03.129743 26641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:03.130048 26641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:03.131219 26641 hybrid_clock.cc:648] HybridClock initialized: now 1790321103131207 us; error 47 us; skew 500 ppm
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:03.133808 26641 init.cc:377] Logged in from keytab as kudu/127.4.47.60@KRBTEST.COM (short username kudu)
I20260925 07:25:03.134274 26641 webserver.cc:460] Webserver started at http://127.4.47.60:34693/ using document root <none> and password file <none>
I20260925 07:25:03.134527 26641 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:03.134598 26641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:03.134755 26641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:03.135994 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data/instance:
uuid: "6850525fbeef460db5a0a077ae91b86b"
format_stamp: "Formatted at 2026-09-25 07:25:03 on dist-test-slave-zw07"
server_key: "a26af9fa6a0592824600392baebeba70"
server_key_iv: "39f0b44f6e9562adb2a8f3b229411d54"
server_key_version: "encryptionkey@0"
I20260925 07:25:03.136363 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal/instance:
uuid: "6850525fbeef460db5a0a077ae91b86b"
format_stamp: "Formatted at 2026-09-25 07:25:03 on dist-test-slave-zw07"
server_key: "a26af9fa6a0592824600392baebeba70"
server_key_iv: "39f0b44f6e9562adb2a8f3b229411d54"
server_key_version: "encryptionkey@0"
I20260925 07:25:03.137800 26641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:25:03.138581 26656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:03.138792 26641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:25:03.138904 26641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "6850525fbeef460db5a0a077ae91b86b"
format_stamp: "Formatted at 2026-09-25 07:25:03 on dist-test-slave-zw07"
server_key: "a26af9fa6a0592824600392baebeba70"
server_key_iv: "39f0b44f6e9562adb2a8f3b229411d54"
server_key_version: "encryptionkey@0"
I20260925 07:25:03.138996 26641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:03.142544 26634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } attempt: 2
I20260925 07:25:03.142949 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.142814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57406)
0925 07:25:03.142872 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.142915 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:25:03.143287 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.143216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57422)
0925 07:25:03.143243 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.143269 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 07:25:03.143396 26634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:34843: Network error: Client connection negotiation failed: client connection to 127.4.47.60:34843: connect: Connection refused (error 111)
I20260925 07:25:03.153406 26641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:03.155941 26641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:03.156095 26641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:03.160279 26641 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.60:34843
I20260925 07:25:03.160356 26708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.60:34843 every 8 connection(s)
I20260925 07:25:03.160722 26641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
I20260925 07:25:03.161780 26709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:03.163398 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:03.170050 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26641
I20260925 07:25:03.170168 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790320837143632-4284-0/minicluster-data/master-2/wal/instance
I20260925 07:25:03.170347 4284 external_mini_cluster.cc:1428] Setting key 8840d3d0402fb8a86c2a13018494905a
I20260925 07:25:03.172546 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.164115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.4.47.62:38939)
0925 07:25:03.164200 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.164203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.164343 (+ 140us) server_negotiation.cc:407] Connection header received
0925 07:25:03.164809 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.164810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.164819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.164872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.165486 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.165973 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.166945 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.167171 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.167223 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.169352 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.169354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.169355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.169368 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.170997 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.171506 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.171507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.171508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.171580 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.171857 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.171858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.171859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.172019 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.172083 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.172409 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.172479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 07:25:03.172780 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.164050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:38939 (local address 127.0.0.1:53902)
0925 07:25:03.164213 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.164258 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.164723 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.164940 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.164943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.165211 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.165410 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.165414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.166056 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.166057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.166867 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.166869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.167252 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.167252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.167363 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.167843 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.167852 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.169245 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.171095 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.171100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.171102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.171420 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.171660 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.171661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.171661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.171783 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.172087 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.172092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.172262 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.172385 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.172544 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":106,"threads_started":1}
I20260925 07:25:03.173729 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:03.180497 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.174011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:34609 (local address 127.0.0.1:54238)
0925 07:25:03.174060 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.174071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.174115 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.174249 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.174250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.174390 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.174435 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.174437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.175106 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.175107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.175607 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.175608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.175991 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.175992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.176064 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.176512 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.176516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.177747 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.179453 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.179454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.179454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.179626 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.179827 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.179828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.179828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.179895 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.180161 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.180162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.180208 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.180292 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.180394 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 07:25:03.180497 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.174098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.4.47.61:34609)
0925 07:25:03.174150 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.174153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.174162 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:03.174188 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.174189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.174198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.174248 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.174524 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.174990 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.175690 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.175901 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.175990 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.177833 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.177834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.177835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.177844 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.179375 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.179698 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.179699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.179700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.179749 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.179961 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.179962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.179962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.180083 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.180157 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.180317 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.180394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:25:03.180853 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.170806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.4.47.60:34843)
0925 07:25:03.170953 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.170958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.171023 (+ 65us) server_negotiation.cc:407] Connection header received
0925 07:25:03.171132 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.171135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.171151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.171234 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.172019 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.173125 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.174106 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.174453 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.174540 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.177099 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.177121 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.177121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.177127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.179373 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.179968 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.179973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.179973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.180035 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.180256 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.180257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.180257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.180410 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.180574 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.180659 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.180738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 07:25:03.181433 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:03.188591 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.181780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.4.47.60:34843)
0925 07:25:03.181811 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.181813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.181822 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:03.181889 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.181890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.181901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.181951 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.182168 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.182744 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.183465 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.183794 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.183854 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.185898 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.185900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.185900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.185907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.187452 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.187806 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.187807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.187807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.187860 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.188046 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.188047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.188048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.188168 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.188236 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.188418 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.188500 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 07:25:03.188596 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.181751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57440)
0925 07:25:03.181788 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.181797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.181831 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.181960 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.181961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.182070 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.182101 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.182102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.182818 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.182819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.183397 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.183399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.183862 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.183863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.183917 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.184374 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.184379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.185813 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.187543 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.187544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.187545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.187722 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.187922 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.187923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.187924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.187976 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.188238 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.188239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.188277 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.188360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.188452 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 07:25:03.189844 26559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } attempt: 2
I20260925 07:25:03.190315 26709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: Bootstrap starting.
I20260925 07:25:03.191394 26709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:03.191505 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.181667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.4.47.62:38939)
0925 07:25:03.181717 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.181719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.181731 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:25:03.181768 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.181769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.181780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.181832 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.182427 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.182948 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.183799 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.184099 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.184191 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.186728 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.186730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.186730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.186738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.188277 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.188644 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.188645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.188646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.188711 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.188933 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.188934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.188935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.189105 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.189274 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.191333 (+ 2059us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.191413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:25:03.191742 26709 log.cc:824] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:03.192878 26709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: No bootstrap required, opened a new log
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:03.194581 26709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.194796 26709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850525fbeef460db5a0a077ae91b86b, State: Initialized, Role: FOLLOWER
I20260925 07:25:03.194896 26709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.195297 26709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:03.195683 26715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.195798 26715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:03.198246 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.190129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57446)
0925 07:25:03.190176 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.190189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.190246 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.190380 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.190381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.190514 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.190550 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.190551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.191539 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.191540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.192319 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.192322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.192792 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.192794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.192857 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.193338 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.193343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.194737 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.196777 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.196779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.196779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.196989 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.197257 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.197258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.197259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.197324 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.197939 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.197940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.198002 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.198051 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.198156 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 07:25:03.198709 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.190161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.4.47.60:34843)
0925 07:25:03.190201 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.190203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.190230 (+ 27us) server_negotiation.cc:407] Connection header received
0925 07:25:03.190302 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.190303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.190312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.190364 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.190623 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.191369 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.192422 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.192694 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.192765 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.194868 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.194869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.194869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.194876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.196650 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.197080 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.197082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.197082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.197145 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.197695 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.197696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.197696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.197856 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.198501 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.198544 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.198623 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 07:25:03.199334 26726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:03.199451 26726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:03.198093 26641 master_runner.cc:186] Error getting master registration for 127.4.47.62:38939: OK, instance_id { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" instance_seqno: 1790321102846071 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:25:03.200807 26641 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:25:03.200800 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: Bootstrap starting.
I20260925 07:25:03.201184 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.192312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.4.47.61:34609)
0925 07:25:03.192354 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.192357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.192364 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:25:03.192390 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.192391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.192402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.192457 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.192979 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.193661 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.194660 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.194942 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.195020 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.197924 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.197925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.197925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.197934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.199875 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.200276 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.200277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.200278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.200336 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.200537 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.200538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.200538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.200671 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.200820 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.201035 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.201102 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:25:03.201556 26559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:03.201931 26559 log.cc:824] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:03.202829 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:25:03.202903 26559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: No bootstrap required, opened a new log
I20260925 07:25:03.203913 26533 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8b726a243c394a1c8d40026c112a0be8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53916
I20260925 07:25:03.205044 26559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.205183 26559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b726a243c394a1c8d40026c112a0be8, State: Initialized, Role: FOLLOWER
I20260925 07:25:03.205268 26559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.205636 26559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:03.205719 26533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 07:25:03.205771 26533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:03.206779 26533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.207044 26533 leader_election.cc:288] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [CANDIDATE]: Term 1 election: Requested vote from peers dd0e6d9803e948b1bb7701908d2e1e11 (127.4.47.61:34609), 6850525fbeef460db5a0a077ae91b86b (127.4.47.60:34843)
I20260925 07:25:03.205580 26729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.207501 26683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b726a243c394a1c8d40026c112a0be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6850525fbeef460db5a0a077ae91b86b"
I20260925 07:25:03.207621 26683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:03.207672 26729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:03.208899 26683 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b726a243c394a1c8d40026c112a0be8 in term 1.
I20260925 07:25:03.209148 26509 leader_election.cc:302] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6850525fbeef460db5a0a077ae91b86b, 8b726a243c394a1c8d40026c112a0be8; no voters:
I20260925 07:25:03.209483 26729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:25:03.209640 26729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [term 1 LEADER]: Becoming Leader. State: Replica: 8b726a243c394a1c8d40026c112a0be8, State: Running, Role: LEADER
I20260925 07:25:03.209734 26729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.210399 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b726a243c394a1c8d40026c112a0be8. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.210597 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:03.210275 26491 master_runner.cc:186] Error getting master registration for 127.4.47.61:34609: OK, instance_id { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" instance_seqno: 1790321102998686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:25:03.212628 26491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 07:25:03.213012 26740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:03.213060 26740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:03.214640 26750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:25:03.214815 26750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:25:03.219115 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.208942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.4.47.60:34843)
0925 07:25:03.209149 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.209150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.210718 (+ 1568us) server_negotiation.cc:407] Connection header received
0925 07:25:03.210829 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.210830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.210841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.210870 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.211615 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.212269 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.213713 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.214000 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.214040 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.215891 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.215892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.215893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.215900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.217831 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.218235 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.218236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.218236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.218289 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.218479 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.218479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.218480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.218599 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.218682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.218883 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.218965 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260925 07:25:03.220072 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.208777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.4.47.62:38939)
0925 07:25:03.208819 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.208822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.209647 (+ 825us) server_negotiation.cc:407] Connection header received
0925 07:25:03.209792 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.209793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.209804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.209834 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.211318 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.212086 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.213555 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.213840 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.213877 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.216718 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.216720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.216720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.216727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.218434 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.218844 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.218846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.218846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.218910 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.219498 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.219499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.219499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.219634 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.219728 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.219909 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.219981 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:25:03.223829 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.209345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.4.47.61:34609)
0925 07:25:03.209398 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.209400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.213688 (+ 4288us) server_negotiation.cc:407] Connection header received
0925 07:25:03.213791 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.213792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.213803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.213832 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.214572 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.215353 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.216963 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.217263 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.217310 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.220221 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.220222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.220223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.220232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.222087 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.222700 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.222701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.222701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.222762 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.223056 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.223057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.223057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.223231 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.223318 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.223654 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.223739 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:25:03.224184 26683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [term 1 FOLLOWER]: Refusing update from remote peer 8b726a243c394a1c8d40026c112a0be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:25:03.224485 26729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:25:03.225850 26715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b726a243c394a1c8d40026c112a0be8. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.225956 26715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:03.230723 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.230859 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:03.231299 26750 catalog_manager.cc:1347] Generated new cluster ID: 79641affdcd1423992fa462665cf4c34
I20260925 07:25:03.231349 26750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:25:03.231626 26715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.231714 26715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:03.231964 26752 mvcc.cc:205] Tried to move back new op lower bound from 7333155238803030016 to 7333155238747729920. Current Snapshot: MvccSnapshot[applied={T|T < 7333155238803030016 or (T in {7333155238803030016})}]
I20260925 07:25:03.232268 26753 mvcc.cc:205] Tried to move back new op lower bound from 7333155238803030016 to 7333155238747729920. Current Snapshot: MvccSnapshot[applied={T|T < 7333155238803030016}]
I20260925 07:25:03.234510 26634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } attempt: 3
Sep 25 07:25:03 dist-test-slave-zw07 krb5kdc[26478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:03.240448 26750 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:25:03.241283 26750 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:25:03.244277 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.234766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:34843 (local address 127.0.0.1:57462)
0925 07:25:03.234813 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:03.234824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:03.234866 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:03.235023 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:03.235024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:03.235260 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:03.235308 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.235310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.236348 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.236350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.237205 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:03.237208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.237791 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.237792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:03.237869 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.238450 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:03.238463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:03.240184 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:03.243158 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.243159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.243160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.243392 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.243600 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:03.243601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:03.243601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:03.243660 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:03.243947 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:03.243948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:03.244001 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 07:25:03.244068 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:25:03.244174 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:25:03.244656 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:03.234809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.4.47.60:34843)
0925 07:25:03.234849 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.234851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.234860 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:03.234940 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.234941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.234952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.235008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.235376 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.236116 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.237336 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:03.237641 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:03.237704 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:03.240268 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:03.240269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:03.240270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:03.240277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:03.243046 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.243474 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.243475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.243475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.243537 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:03.243733 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:03.243735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:03.243735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:03.243868 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:03.243928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:03.244486 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 07:25:03.244566 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:25:03.246183 26634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: Bootstrap starting.
I20260925 07:25:03.246954 26634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:03.247314 26634 log.cc:824] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:03.251919 26634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: No bootstrap required, opened a new log
I20260925 07:25:03.254024 26634 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.254184 26634 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0e6d9803e948b1bb7701908d2e1e11, State: Initialized, Role: FOLLOWER
I20260925 07:25:03.254273 26634 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } }
I20260925 07:25:03.254642 26634 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:03.256137 26607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:03.257083 26607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [term 1 FOLLOWER]: Refusing update from remote peer 8b726a243c394a1c8d40026c112a0be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:25:03.257391 26729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:25:03.255981 26608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b726a243c394a1c8d40026c112a0be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dd0e6d9803e948b1bb7701908d2e1e11"
I20260925 07:25:03.258379 26608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b726a243c394a1c8d40026c112a0be8 in term 1.
I20260925 07:25:03.259948 26773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b726a243c394a1c8d40026c112a0be8. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.260114 26773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:03.260623 26773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.260763 26773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:03.263357 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.263466 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:25:03.264082 26769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:03.264192 26769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:03.265328 26729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b726a243c394a1c8d40026c112a0be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b726a243c394a1c8d40026c112a0be8" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 38939 } } peers { permanent_uuid: "dd0e6d9803e948b1bb7701908d2e1e11" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 34609 } } peers { permanent_uuid: "6850525fbeef460db5a0a077ae91b86b" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 34843 } } }
I20260925 07:25:03.265425 26729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:03.266333 26750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b726a243c394a1c8d40026c112a0be8: Generated new TSK 0
I20260925 07:25:03.266526 26750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:25:03.267720 26771 mvcc.cc:205] Tried to move back new op lower bound from 7333155238855057408 to 7333155238747729920. Current Snapshot: MvccSnapshot[applied={T|T < 7333155238803030016}]
W20260925 07:25:03.291811 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.290155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.4.47.61:34609)
0925 07:25:03.290203 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.290206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.291082 (+ 876us) server_negotiation.cc:407] Connection header received
0925 07:25:03.291203 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.291204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.291217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.291248 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.291734 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54254: BlockingRecv error: recv got EOF from 127.0.0.1:54254 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 07:25:03.292593 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.290585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.4.47.62:38939)
0925 07:25:03.290630 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.290632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.292061 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 07:25:03.292175 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.292176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.292187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.292215 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.292532 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53934: BlockingRecv error: recv got EOF from 127.0.0.1:53934 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:25:03.293430 26713 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.290954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.4.47.60:34843)
0925 07:25:03.291014 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.291017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.292809 (+ 1792us) server_negotiation.cc:407] Connection header received
0925 07:25:03.292980 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.292981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.292992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.293022 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.293375 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57466: BlockingRecv error: recv got EOF from 127.0.0.1:57466 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 07:25:03.296144 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.295329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.4.47.61:34609)
0925 07:25:03.295365 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.295366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.295569 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:25:03.295683 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.295684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.295703 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.295733 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.296064 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54270: BlockingRecv error: recv got EOF from 127.0.0.1:54270 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 07:25:03.299281 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.298476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.4.47.61:34609)
0925 07:25:03.298509 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.298511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.298731 (+ 220us) server_negotiation.cc:407] Connection header received
0925 07:25:03.298827 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.298828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.298839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.298866 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.299231 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54280: BlockingRecv error: recv got EOF from 127.0.0.1:54280 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 07:25:03.304483 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.303237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.4.47.61:34609)
0925 07:25:03.303297 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.303299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.303662 (+ 363us) server_negotiation.cc:407] Connection header received
0925 07:25:03.303803 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.303805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.303842 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.303872 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.304429 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54292: BlockingRecv error: recv got EOF from 127.0.0.1:54292 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 07:25:03.324055 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.315480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.4.47.61:34609)
0925 07:25:03.315529 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.315531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.323336 (+ 7805us) server_negotiation.cc:407] Connection header received
0925 07:25:03.323466 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.323468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.323482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.323515 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.323987 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54306: BlockingRecv error: recv got EOF from 127.0.0.1:54306 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 07:25:03.334506 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.333532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.4.47.61:34609)
0925 07:25:03.333594 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.333596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.333881 (+ 285us) server_negotiation.cc:407] Connection header received
0925 07:25:03.333990 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.333991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.334004 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.334036 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.334429 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54312: BlockingRecv error: recv got EOF from 127.0.0.1:54312 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:25:03.353034 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.352052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.4.47.61:34609)
0925 07:25:03.352101 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.352103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.352386 (+ 283us) server_negotiation.cc:407] Connection header received
0925 07:25:03.352498 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.352499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.352513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.352558 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.352970 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54324: BlockingRecv error: recv got EOF from 127.0.0.1:54324 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:25:03.387706 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.386493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.4.47.61:34609)
0925 07:25:03.386550 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.386553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.386922 (+ 369us) server_negotiation.cc:407] Connection header received
0925 07:25:03.387052 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.387054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.387094 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.387135 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.387639 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54336: BlockingRecv error: recv got EOF from 127.0.0.1:54336 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 07:25:03.453855 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.453157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.4.47.61:34609)
0925 07:25:03.453257 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.453259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.453335 (+ 76us) server_negotiation.cc:407] Connection header received
0925 07:25:03.453367 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.453368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.453378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.453444 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.453778 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54352: BlockingRecv error: recv got EOF from 127.0.0.1:54352 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 07:25:03.583738 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.583055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.4.47.61:34609)
0925 07:25:03.583141 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.583143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.583198 (+ 55us) server_negotiation.cc:407] Connection header received
0925 07:25:03.583254 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.583255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.583266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.583318 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.583674 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54360: BlockingRecv error: recv got EOF from 127.0.0.1:54360 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 07:25:03.841651 26638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:03.840892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.4.47.61:34609)
0925 07:25:03.840983 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:03.840985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:03.841004 (+ 19us) server_negotiation.cc:407] Connection header received
0925 07:25:03.841031 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:03.841031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:03.841041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:03.841104 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:03.841585 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54372: BlockingRecv error: recv got EOF from 127.0.0.1:54372 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 07:25:04.200045 26726 catalog_manager.cc:1259] Loaded cluster ID: 79641affdcd1423992fa462665cf4c34
I20260925 07:25:04.200133 26726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: loading cluster ID for follower catalog manager: success
I20260925 07:25:04.201051 26726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: acquiring CA information for follower catalog manager: success
I20260925 07:25:04.201390 26726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6850525fbeef460db5a0a077ae91b86b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:25:04.264710 26769 catalog_manager.cc:1259] Loaded cluster ID: 79641affdcd1423992fa462665cf4c34
I20260925 07:25:04.264793 26769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: loading cluster ID for follower catalog manager: success
I20260925 07:25:04.265617 26769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: acquiring CA information for follower catalog manager: success
I20260925 07:25:04.265902 26769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd0e6d9803e948b1bb7701908d2e1e11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:25:04.357909 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:04.354916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.4.47.61:34609)
0925 07:25:04.355051 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:04.355054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:04.355157 (+ 103us) server_negotiation.cc:407] Connection header received
0925 07:25:04.355267 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:04.355269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:04.355334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:04.355435 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:04.355660 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.356237 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.357048 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.357261 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.357337 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:04.357542 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:04.357657 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:04.357749 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:04.357772 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 07:25:04.357796 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260925 07:25:04.362623 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:04.360047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.4.47.60:34843)
0925 07:25:04.360177 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:04.360179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:04.360188 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:04.360255 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:04.360255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:04.360309 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:04.360400 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:04.360521 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.361013 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.361766 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.361982 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.362097 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:04.362263 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:04.362382 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:04.362473 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:04.362501 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:25:04.362523 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260925 07:25:04.367267 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:04.364697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.4.47.60:34843)
0925 07:25:04.364759 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:04.364761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:04.365064 (+ 303us) server_negotiation.cc:407] Connection header received
0925 07:25:04.365146 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:04.365147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:04.365157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:04.365208 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:04.365307 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.365776 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.366555 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.366750 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.366845 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:04.366997 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:04.367029 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:04.367107 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:04.367167 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:25:04.367189 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:25:04.367837 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:04.364406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.4.47.61:34609)
0925 07:25:04.364449 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:04.364451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:04.364755 (+ 304us) server_negotiation.cc:407] Connection header received
0925 07:25:04.364816 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:04.364817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:04.364827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:04.364882 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:04.364980 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.365750 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.366937 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.367268 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.367328 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:04.367532 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:04.367566 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:04.367591 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:04.367717 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 07:25:04.367745 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:25:04.368237 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:04.364404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.4.47.62:38939)
0925 07:25:04.364712 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:04.364732 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:04.364742 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:25:04.364788 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:04.364789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:04.364844 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:04.364937 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:04.365099 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.365832 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.366916 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:04.367246 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:04.367303 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:04.367539 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:04.367656 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:04.367719 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:04.368094 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 07:25:04.368118 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":53,"threads_started":1}
I20260925 07:25:04.369817 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26491
I20260925 07:25:04.375324 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26566
I20260925 07:25:04.380163 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26641
2026-09-25T07:25:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2800 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:25:04 dist-test-slave-zw07 krb5kdc[26859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:25:04 dist-test-slave-zw07 krb5kdc[26859](info): set up 2 sockets
Sep 25 07:25:04 dist-test-slave-zw07 krb5kdc[26859](info): commencing operation
krb5kdc: starting...
W20260925 07:25:05.427392 4284 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/test-user.keytab.
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.47.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
Entry for principal kudu/127.4.47.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab.
I20260925 07:25:05.546063 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37599
--webserver_interface=127.4.47.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:25:05.627700 26872 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:05.627993 26872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:05.628026 26872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:05.628073 26872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:05.629321 26872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:05.629406 26872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:05.629431 26872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:05.629453 26872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:05.629482 26872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:05.631119 26872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.62.keytab
--principal=kudu/127.4.47.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.62:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.47.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:05.631429 26872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:05.631699 26872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:05.636770 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:05.636776 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:05.636786 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:05.636986 26872 server_base.cc:1034] running on GCE node
I20260925 07:25:05.637327 26872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:05.637595 26872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:05.638767 26872 hybrid_clock.cc:648] HybridClock initialized: now 1790321105638745 us; error 43 us; skew 500 ppm
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:05.647917 26872 init.cc:377] Logged in from keytab as kudu/127.4.47.62@KRBTEST.COM (short username kudu)
I20260925 07:25:05.648550 26872 webserver.cc:460] Webserver started at http://127.4.47.62:45785/ using document root <none> and password file <none>
I20260925 07:25:05.648897 26872 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:05.648979 26872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:05.649173 26872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:05.650434 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data/instance:
uuid: "9c1d0297fe024d788c848986bc3d0471"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "eaa9fda5807264500902b143eb297573"
server_key_iv: "d149f65c9cef38291df509af1ce4f667"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.650840 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance:
uuid: "9c1d0297fe024d788c848986bc3d0471"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "eaa9fda5807264500902b143eb297573"
server_key_iv: "d149f65c9cef38291df509af1ce4f667"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.652316 26872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:25:05.653170 26887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:05.653362 26872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:05.653465 26872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal
uuid: "9c1d0297fe024d788c848986bc3d0471"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "eaa9fda5807264500902b143eb297573"
server_key_iv: "d149f65c9cef38291df509af1ce4f667"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.653542 26872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:05.662995 26872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:05.665217 26872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:05.665342 26872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:05.669580 26872 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.62:37599
I20260925 07:25:05.669618 26939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.62:37599 every 8 connection(s)
I20260925 07:25:05.670009 26872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/data/info.pb
I20260925 07:25:05.671008 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26872
I20260925 07:25:05.671151 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-0/wal/instance
I20260925 07:25:05.671197 26940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:05.671379 4284 external_mini_cluster.cc:1428] Setting key c083d78faa584e7a23289b69c1035f59
I20260925 07:25:05.673547 26940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.47.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.61@KRBTEST.COM" created.
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:05.685973 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.674380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.4.47.62:37599)
0925 07:25:05.674575 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:05.674586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:05.674714 (+ 128us) server_negotiation.cc:407] Connection header received
0925 07:25:05.675423 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:05.675425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:05.675451 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:05.675556 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:05.676172 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.676995 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.677902 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.678226 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.678373 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.682461 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:05.682470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:05.682470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:05.682479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:05.684249 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.684676 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.684680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.684680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.684768 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.684999 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.685000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.685000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.685237 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:05.685335 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:05.685517 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 07:25:05.685700 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260925 07:25:05.685976 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.674347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37599 (local address 127.0.0.1:60024)
0925 07:25:05.674556 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.674600 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:05.675252 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:05.675561 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:05.675563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:05.675867 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:05.676044 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.676049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.677107 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.677108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.677799 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.677801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.678351 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.678353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.678416 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.679050 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:05.679059 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:05.682327 (+ 3268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:05.684362 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.684369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.684371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.684574 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.684841 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.684842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.684842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.684931 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.685311 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:05.685316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:05.685379 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 07:25:05.685493 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 07:25:05.685749 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":117,"thread_start_us":117,"threads_started":1}
I20260925 07:25:05.686772 26940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:05.687194 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.687038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36774)
0925 07:25:05.687115 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.687162 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 07:25:05.687372 26890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
I20260925 07:25:05.687852 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.687758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36786)
0925 07:25:05.687811 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.687836 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 07:25:05.687981 26940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:36245: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
Entry for principal kudu/127.4.47.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab.
I20260925 07:25:05.698428 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:36245
--webserver_interface=127.4.47.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:25:05.737236 26940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } attempt: 1
I20260925 07:25:05.737846 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.737699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36800)
0925 07:25:05.737771 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.737815 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 07:25:05.738278 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.738193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36812)
0925 07:25:05.738236 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.738262 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:25:05.738384 26940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:36245: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
W20260925 07:25:05.782953 26947 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:05.783219 26947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:05.783237 26947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:05.783258 26947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:05.784482 26947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:05.784533 26947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:05.784549 26947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:05.784561 26947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:05.784574 26947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:05.786132 26947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.61.keytab
--principal=kudu/127.4.47.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.61:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.47.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:05.786398 26947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:05.786636 26947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:05.791757 26955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:05.791885 26947 server_base.cc:1034] running on GCE node
W20260925 07:25:05.791747 26953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:05.791742 26952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:05.792198 26947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:05.792450 26947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:05.793596 26947 hybrid_clock.cc:648] HybridClock initialized: now 1790321105793577 us; error 41 us; skew 500 ppm
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:05.796172 26947 init.cc:377] Logged in from keytab as kudu/127.4.47.61@KRBTEST.COM (short username kudu)
I20260925 07:25:05.796576 26947 webserver.cc:460] Webserver started at http://127.4.47.61:42991/ using document root <none> and password file <none>
I20260925 07:25:05.796765 26947 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:05.796806 26947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:05.796897 26947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:05.797967 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data/instance:
uuid: "e6d3298e6f504d13a5c73434436f6ef3"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "861fdc103b593e54f2fe65b2c287fd22"
server_key_iv: "21a9ea0c89829f70cb1389050c7c615e"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.798281 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal/instance:
uuid: "e6d3298e6f504d13a5c73434436f6ef3"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "861fdc103b593e54f2fe65b2c287fd22"
server_key_iv: "21a9ea0c89829f70cb1389050c7c615e"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.799731 26947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 07:25:05.800482 26962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:05.800743 26947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:05.800843 26947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal
uuid: "e6d3298e6f504d13a5c73434436f6ef3"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "861fdc103b593e54f2fe65b2c287fd22"
server_key_iv: "21a9ea0c89829f70cb1389050c7c615e"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.800930 26947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:05.806528 26940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } attempt: 2
I20260925 07:25:05.806972 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.806823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36816)
0925 07:25:05.806887 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.806934 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 07:25:05.807371 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.807283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36832)
0925 07:25:05.807326 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.807352 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:25:05.807545 26940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.61:36245: Network error: Client connection negotiation failed: client connection to 127.4.47.61:36245: connect: Connection refused (error 111)
I20260925 07:25:05.826968 26947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:05.829030 26947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:05.829185 26947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:05.833484 26947 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.61:36245
I20260925 07:25:05.833561 27014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.61:36245 every 8 connection(s)
I20260925 07:25:05.833918 26947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/data/info.pb
I20260925 07:25:05.834964 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:05.836632 27015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:05.843938 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 26947
I20260925 07:25:05.844081 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-1/wal/instance
I20260925 07:25:05.844285 4284 external_mini_cluster.cc:1428] Setting key ac35f63a1173147ed8d44f98e8add708
I20260925 07:25:05.846037 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.837369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.4.47.62:37599)
0925 07:25:05.837451 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:05.837453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:05.837633 (+ 180us) server_negotiation.cc:407] Connection header received
0925 07:25:05.838071 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:05.838072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:05.838085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:05.838140 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:05.838750 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.839262 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.840276 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.840493 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.840552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.842720 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:05.842722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:05.842722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:05.842729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:05.844366 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.844892 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.844893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.844894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.844948 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.845236 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.845236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.845237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.845357 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:05.845412 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:05.845784 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 07:25:05.845889 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:25:05.846179 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.837320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37599 (local address 127.0.0.1:60030)
0925 07:25:05.837498 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.837537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:05.837987 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:05.838210 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:05.838213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:05.838467 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:05.838672 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.838676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.839344 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.839345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.840187 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.840190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.840583 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.840584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.840700 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.841219 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:05.841228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:05.842629 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:05.844468 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.844476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.844479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.844812 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.845017 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.845018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.845018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.845166 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.845426 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:05.845432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:05.845627 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 07:25:05.845752 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 07:25:05.845959 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260925 07:25:05.846777 27015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.47.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.47.60@KRBTEST.COM" created.
I20260925 07:25:05.855818 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.847155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.4.47.61:36245)
0925 07:25:05.847339 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:05.847344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:05.847368 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:25:05.847392 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:05.847394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:05.847409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:05.847489 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:05.847745 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.848985 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.849719 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.850359 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.850451 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.852609 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:05.852620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:05.852620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:05.852628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:05.854397 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.854763 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.854767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.854767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.854826 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.855100 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.855101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.855102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.855322 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:05.855407 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:05.855588 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 07:25:05.855685 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260925 07:25:05.855901 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.847112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36842)
0925 07:25:05.847161 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.847173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:05.847213 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:05.847504 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:05.847505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:05.847622 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:05.847672 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.847673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.849081 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.849083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.849623 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.849625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.850459 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.850460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.850526 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.850998 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:05.851003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:05.852508 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:05.854504 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.854505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.854505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.854680 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.854919 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.854920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.854920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.855005 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.855382 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:05.855383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:05.855452 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 07:25:05.855555 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 07:25:05.855777 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":118}
I20260925 07:25:05.856606 27015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:05.857015 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.856869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33126)
0925 07:25:05.856921 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.856978 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:25:05.857170 26965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
I20260925 07:25:05.857666 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.857568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33130)
0925 07:25:05.857622 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.857648 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:25:05.857772 27015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:44923: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
Entry for principal kudu/127.4.47.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab.
I20260925 07:25:05.872118 4284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSRTLta/build/release/bin/kudu
/tmp/dist-test-taskSRTLta/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:44923
--webserver_interface=127.4.47.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.47.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:25:05.898744 26940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } attempt: 3
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:05.906416 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.899232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36844)
0925 07:25:05.899303 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.899315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:05.899386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:05.899539 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:05.899540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:05.899708 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:05.899771 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.899773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.900516 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.900517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.901057 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:05.901059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.901477 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.901478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:05.901557 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.902109 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:05.902114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:05.903641 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:05.905338 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.905339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.905340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.905519 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.905755 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:05.905756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:05.905756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:05.905822 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:05.906071 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:05.906072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:05.906120 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 07:25:05.906206 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:25:05.906314 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:25:05.906416 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.899313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.4.47.61:36245)
0925 07:25:05.899388 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:05.899390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:05.899401 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:25:05.899463 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:05.899464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:05.899477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:05.899534 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:05.899861 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.900431 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.901152 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:05.901390 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:05.901467 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:05.903741 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:05.903743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:05.903744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:05.903752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:05.905229 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.905593 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.905595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.905595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.905657 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:05.905877 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:05.905878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:05.905878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:05.905996 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:05.906069 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:05.906231 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:25:05.906314 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:25:05.906780 26940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:25:05.906994 27015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } attempt: 1
I20260925 07:25:05.907223 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.907041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33138)
0925 07:25:05.907117 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.907188 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 07:25:05.907388 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.907255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33140)
0925 07:25:05.907313 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.907354 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:25:05.907524 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.907443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33156)
0925 07:25:05.907476 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.907501 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 07:25:05.907642 26940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:44923: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
I20260925 07:25:05.907738 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.907650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33168)
0925 07:25:05.907685 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.907715 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 07:25:05.907845 27015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:44923: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
I20260925 07:25:05.938885 26940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } attempt: 1
I20260925 07:25:05.939495 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.939336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33182)
0925 07:25:05.939420 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.939464 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 07:25:05.939908 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.939820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33194)
0925 07:25:05.939864 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.939891 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:25:05.940029 26940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:44923: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
W20260925 07:25:05.955860 27022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:25:05.956123 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:25:05.956156 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:25:05.956179 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:25:05.957448 27022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:25:05.957507 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:25:05.957523 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:25:05.957535 27022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:25:05.957548 27022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:25:05.959031 27022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.47.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.47.62:37599,127.4.47.61:36245,127.4.47.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/krb5kdc/kudu_127.4.47.60.keytab
--principal=kudu/127.4.47.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.47.60:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.47.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:25:05.959311 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:25:05.959527 27022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:25:05.964561 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:25:05.964576 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:05.964668 27022 server_base.cc:1034] running on GCE node
W20260925 07:25:05.964576 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:25:05.965037 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:25:05.965298 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:25:05.966454 27022 hybrid_clock.cc:648] HybridClock initialized: now 1790321105966436 us; error 33 us; skew 500 ppm
Sep 25 07:25:05 dist-test-slave-zw07 krb5kdc[26859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:25:05.969044 27022 init.cc:377] Logged in from keytab as kudu/127.4.47.60@KRBTEST.COM (short username kudu)
I20260925 07:25:05.969581 27022 webserver.cc:460] Webserver started at http://127.4.47.60:44531/ using document root <none> and password file <none>
I20260925 07:25:05.969823 27022 fs_manager.cc:359] Metadata directory not provided
I20260925 07:25:05.969869 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:25:05.970039 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:25:05.971242 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data/instance:
uuid: "f73e555988644e3ca77773c962e15c2e"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "203279611723b5443bd4816c40300526"
server_key_iv: "22caadcf8756ee7049937bb7f3b16f97"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.971650 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal/instance:
uuid: "f73e555988644e3ca77773c962e15c2e"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "203279611723b5443bd4816c40300526"
server_key_iv: "22caadcf8756ee7049937bb7f3b16f97"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.973095 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:05.974011 27037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:25:05.974323 27022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:25:05.974411 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data,/tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal
uuid: "f73e555988644e3ca77773c962e15c2e"
format_stamp: "Formatted at 2026-09-25 07:25:05 on dist-test-slave-zw07"
server_key: "203279611723b5443bd4816c40300526"
server_key_iv: "22caadcf8756ee7049937bb7f3b16f97"
server_key_version: "encryptionkey@0"
I20260925 07:25:05.974572 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:25:05.975971 27015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } attempt: 2
I20260925 07:25:05.976385 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.976266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33198)
0925 07:25:05.976322 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.976358 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:25:05.976728 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:05.976634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33206)
0925 07:25:05.976679 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:05.976704 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 07:25:05.976830 27015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.47.60:44923: Network error: Client connection negotiation failed: client connection to 127.4.47.60:44923: connect: Connection refused (error 111)
I20260925 07:25:05.990731 27022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:25:05.992846 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:25:05.993017 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:25:05.997275 27022 rpc_server.cc:307] RPC server started. Bound to: 127.4.47.60:44923
I20260925 07:25:05.997342 27089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.47.60:44923 every 8 connection(s)
I20260925 07:25:05.997697 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/data/info.pb
I20260925 07:25:05.998708 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:25:06.000494 27090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:06.007354 4284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSRTLta/build/release/bin/kudu as pid 27022
I20260925 07:25:06.007484 4284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSRTLta/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790320837143632-4284-0/minicluster-data/master-2/wal/instance
I20260925 07:25:06.007686 4284 external_mini_cluster.cc:1428] Setting key 0a18534b3d099f6e11feab466a1a2f0c
I20260925 07:25:06.011615 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.001312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.62:37599 (local address 127.0.0.1:60042)
0925 07:25:06.001527 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:06.001564 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:06.002037 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:06.002263 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:06.002266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:06.002534 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:06.002743 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.002747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.003604 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.003605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.004457 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.004460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.004874 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.004875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.005035 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.005645 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:06.005655 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:06.007266 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:06.009968 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.009977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.009980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.010290 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.010515 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.010515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.010516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.010672 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.010908 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:06.010914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:06.011117 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 07:25:06.011234 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 07:25:06.011392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260925 07:25:06.012125 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.001331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.4.47.62:37599)
0925 07:25:06.001401 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.001403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.001661 (+ 258us) server_negotiation.cc:407] Connection header received
0925 07:25:06.002136 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.002137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.002148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.002207 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.002835 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.003511 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.004545 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.004738 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.004823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.007394 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.007395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.007396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.007408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.009875 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.010377 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.010379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.010379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.010444 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.010738 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.010739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.010739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.010845 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.010895 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.011918 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.012021 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:25:06.012633 27090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:06.015349 26940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } attempt: 2
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:06.017787 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.008103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.4.47.60:44923)
0925 07:25:06.008287 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.008291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.008383 (+ 92us) server_negotiation.cc:407] Connection header received
0925 07:25:06.008436 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.008439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.008454 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.008535 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.009442 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.010153 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.011219 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.011531 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.011590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.014250 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.014258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.014258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.014264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.016363 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.016763 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.016766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.016767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.016828 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.017071 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.017072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.017072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.017267 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.017424 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.017486 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.017563 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260925 07:25:06.021137 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.012950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.61:36245 (local address 127.0.0.1:36858)
0925 07:25:06.013000 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:06.013011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:06.013049 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:06.013213 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:06.013213 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:06.013326 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:06.013375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.013376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.014129 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.014130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.014910 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.014912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.015516 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.015517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.015601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.015971 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:06.015977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:06.017513 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:06.019878 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.019879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.019880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.020093 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.020424 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.020425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.020425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.020487 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.020802 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:06.020803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:06.020850 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:25:06.020926 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:25:06.021019 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:25:06.021176 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.013012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.4.47.61:36245)
0925 07:25:06.013054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.013056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.013072 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:25:06.013123 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.013124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.013135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.013197 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.013451 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.014046 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.014999 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.015392 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.015475 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.017617 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.017619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.017620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.017627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.019802 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.020258 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.020260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.020260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.020335 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.020576 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.020577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.020578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.020718 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.020791 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.020965 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.021056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.62@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:06.021790 27090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.62@KRBTEST.COM
I20260925 07:25:06.025823 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.015647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33220)
0925 07:25:06.015715 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:06.015728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:06.015772 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:06.016153 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:06.016154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:06.016321 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:06.016368 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.016370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.017527 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.017528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.018250 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.018252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.018942 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.018943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.019262 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.020261 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:06.020265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:06.021734 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:06.024628 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.024629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.024629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.024892 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.025134 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.025135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.025135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.025203 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.025458 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:06.025459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:06.025514 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 07:25:06.025606 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 07:25:06.025711 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 07:25:06.025858 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.015843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.4.47.60:44923)
0925 07:25:06.016009 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.016011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.016020 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:06.016040 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.016040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.016052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.016126 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.016759 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.017426 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.018368 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.018710 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.018943 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.021830 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.021832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.021833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.021841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.024524 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.024976 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.024977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.024977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.025051 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.025271 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.025272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.025273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.025388 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.025466 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.025632 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.025715 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260925 07:25:06.026790 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.018713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.4.47.62:37599)
0925 07:25:06.018786 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.018788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.018794 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:25:06.018812 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.018813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.018821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.018870 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.019536 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.020157 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.021170 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.021410 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.021490 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.023905 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.023906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.023907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.023913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.025574 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.025961 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.025963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.025963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.026028 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.026239 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.026240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.026241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.026365 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.026501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.026601 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.026683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:25:06.027944 26940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: Bootstrap starting.
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.60@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:06.028798 26940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:06.029161 26940 log.cc:824] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:06.030128 26940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: No bootstrap required, opened a new log
I20260925 07:25:06.031267 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.023057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33222)
0925 07:25:06.023123 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:06.023134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:06.023182 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:06.023517 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:06.023518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:06.023703 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:06.023736 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.023737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.024612 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.024613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.025420 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.025433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.026349 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.026350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.026410 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.026768 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:06.026774 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:06.028294 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:06.030105 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.030106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.030106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.030305 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.030535 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.030536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.030536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.030591 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.030879 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:06.030880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:06.030920 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:25:06.031015 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:25:06.031157 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 07:25:06.031309 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.023112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.4.47.60:44923)
0925 07:25:06.023144 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.023146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.023169 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:25:06.023236 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.023237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.023247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.023303 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.023814 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.024517 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.025527 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.025840 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.026350 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.028389 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.028390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.028391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.028398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.030002 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.030399 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.030400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.030401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.030463 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.030663 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.030664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.030664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.030810 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.030888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.031050 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.031207 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":18}
I20260925 07:25:06.032454 26940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.032614 26940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1d0297fe024d788c848986bc3d0471, State: Initialized, Role: FOLLOWER
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.47.61@KRBTEST.COM
I20260925 07:25:06.032706 26940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.033011 27097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.033167 27097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.033128 27090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: Bootstrap starting.
I20260925 07:25:06.033094 26940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:06.033946 27090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:06.034251 27090 log.cc:824] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:06.035259 27090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: No bootstrap required, opened a new log
I20260925 07:25:06.036381 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.027452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.4.47.61:36245)
0925 07:25:06.027522 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.027524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.027567 (+ 43us) server_negotiation.cc:407] Connection header received
0925 07:25:06.027642 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.027643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.027653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.027709 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.028233 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.028944 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.030113 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.030402 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.030450 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.033190 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.033192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.033192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.033201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.034858 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.035314 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.035315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.035316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.035382 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.035710 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.035712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.035712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.035847 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.035983 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.036202 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.036291 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 07:25:06.036577 27109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:06.036630 27109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:06.037650 27090 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.037882 27090 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73e555988644e3ca77773c962e15c2e, State: Initialized, Role: FOLLOWER
I20260925 07:25:06.038151 27090 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.038288 4284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:25:06.038492 27090 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:06.039273 26914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9c1d0297fe024d788c848986bc3d0471"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60058
I20260925 07:25:06.039428 26914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 07:25:06.039480 26914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:06.039894 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.039988 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.040400 26914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
W20260925 07:25:06.040457 27122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:06.040577 27122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:06.037477 26872 master_runner.cc:186] Error getting master registration for 127.4.47.61:36245: OK, instance_id { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" instance_seqno: 1790321105832191 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:25:06.040634 26872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:25:06.040935 27064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1d0297fe024d788c848986bc3d0471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f73e555988644e3ca77773c962e15c2e"
I20260925 07:25:06.041056 27064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:06.040643 26914 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [CANDIDATE]: Term 1 election: Requested vote from peers e6d3298e6f504d13a5c73434436f6ef3 (127.4.47.61:36245), f73e555988644e3ca77773c962e15c2e (127.4.47.60:44923)
I20260925 07:25:06.041917 27064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d0297fe024d788c848986bc3d0471 in term 1.
I20260925 07:25:06.042363 26890 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c1d0297fe024d788c848986bc3d0471, f73e555988644e3ca77773c962e15c2e; no voters:
I20260925 07:25:06.042574 27097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:25:06.042733 27097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1d0297fe024d788c848986bc3d0471, State: Running, Role: LEADER
I20260925 07:25:06.042824 27097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.043322 27130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1d0297fe024d788c848986bc3d0471. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.043469 27130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:06.043890 27022 master_runner.cc:186] Error getting master registration for 127.4.47.61:36245: OK, instance_id { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" instance_seqno: 1790321105832191 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:25:06.046296 27022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:25:06.057533 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.043316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.4.47.60:44923)
0925 07:25:06.043358 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.043360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.043839 (+ 479us) server_negotiation.cc:407] Connection header received
0925 07:25:06.043987 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.043988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.043999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.044031 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.047556 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.048214 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.050221 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.050518 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.050564 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.052862 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.052864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.052864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.052873 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.054720 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.056328 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.056330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.056330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.056392 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.056715 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.056728 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.056728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.056879 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.056981 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.057322 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.057413 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:25:06.064431 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.047289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.4.47.62:37599)
0925 07:25:06.047341 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.047344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.055232 (+ 7888us) server_negotiation.cc:407] Connection header received
0925 07:25:06.055417 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.055419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.055439 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.055554 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.056510 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.057286 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.059088 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.059396 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.059467 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.061087 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.061088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.061088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.061096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.062822 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.063403 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.063405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.063405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.063476 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.063691 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.063691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.063692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.063815 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.063901 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.064241 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.064324 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:25:06.064407 27132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:25:06.064980 27132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:25:06.065694 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.042204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.4.47.61:36245)
0925 07:25:06.042246 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.042249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.043744 (+ 1495us) server_negotiation.cc:407] Connection header received
0925 07:25:06.044869 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.044870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.044883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.044914 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.045656 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.046402 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.049280 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.049704 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.049762 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.053180 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.053182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.053182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.053191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.055519 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.060060 (+ 4541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.060062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.060063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.060133 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.061878 (+ 1745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.061879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.061880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.062043 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.062150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.065491 (+ 3341us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.065595 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:25:06.067595 27064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [term 1 FOLLOWER]: Refusing update from remote peer 9c1d0297fe024d788c848986bc3d0471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:25:06.067955 27015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } attempt: 3
I20260925 07:25:06.067977 27130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:25:06.069454 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1d0297fe024d788c848986bc3d0471. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.069547 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.070829 27097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.070927 27097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:06.071405 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.071498 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.071861 27134 mvcc.cc:205] Tried to move back new op lower bound from 7333155250449162240 to 7333155250352144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333155250449162240}]
I20260925 07:25:06.072412 27135 mvcc.cc:205] Tried to move back new op lower bound from 7333155250449162240 to 7333155250352144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333155250449162240}]
I20260925 07:25:06.072973 27132 catalog_manager.cc:1347] Generated new cluster ID: 15b2c17f56e8494981cccaae4f9aade2
I20260925 07:25:06.073019 27132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:25:06 dist-test-slave-zw07 krb5kdc[26859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.47.61@KRBTEST.COM for kudu/127.4.47.60@KRBTEST.COM
I20260925 07:25:06.077837 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.068208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.47.60:44923 (local address 127.0.0.1:33252)
0925 07:25:06.068294 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 07:25:06.068305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:25:06.068341 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:25:06.068618 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:25:06.068619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:25:06.068799 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:25:06.068847 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.068849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.069818 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.069819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.070664 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:25:06.070665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.071240 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.071242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:25:06.071313 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.071881 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:25:06.071887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:25:06.073505 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:25:06.076308 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.076309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.076310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.076539 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.076841 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:25:06.076842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:25:06.076842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:25:06.076913 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:25:06.077415 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:25:06.077415 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:25:06.077508 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 07:25:06.077612 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:25:06.077727 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 07:25:06.078603 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:06.068409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.4.47.60:44923)
0925 07:25:06.068472 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.068474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.068481 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:25:06.068513 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.068514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.068524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.068571 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.068939 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.069710 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.070776 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:06.071111 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:06.071173 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:06.073618 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:25:06.073620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:25:06.073620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:25:06.073627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:25:06.076134 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.076639 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.076640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.076640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.076755 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:25:06.076998 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:25:06.076998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:25:06.076999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:25:06.077282 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:25:06.078120 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:06.078207 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 07:25:06.078346 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:25:06.080322 27015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: Bootstrap starting.
I20260925 07:25:06.081264 27015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: Neither blocks nor log segments found. Creating new log.
I20260925 07:25:06.081598 27015 log.cc:824] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: Log is configured to *not* fsync() on all Append() calls
I20260925 07:25:06.087823 27015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: No bootstrap required, opened a new log
I20260925 07:25:06.089927 27015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.090096 27015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d3298e6f504d13a5c73434436f6ef3, State: Initialized, Role: FOLLOWER
I20260925 07:25:06.090181 27015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } }
I20260925 07:25:06.090544 27015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:25:06.090687 27132 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:25:06.091527 27132 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:25:06.092010 26989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1d0297fe024d788c848986bc3d0471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e6d3298e6f504d13a5c73434436f6ef3"
I20260925 07:25:06.091938 26988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:25:06.094295 26988 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [term 1 FOLLOWER]: Refusing update from remote peer 9c1d0297fe024d788c848986bc3d0471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:25:06.094676 27138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:25:06.095299 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.095394 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.096904 26989 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d0297fe024d788c848986bc3d0471 in term 1.
I20260925 07:25:06.099146 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1d0297fe024d788c848986bc3d0471. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.099254 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:25:06.099344 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.099395 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:25:06.099807 27152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:25:06.099862 27152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:25:06.100786 27153 mvcc.cc:205] Tried to move back new op lower bound from 7333155250540449792 to 7333155250352144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333155250449162240}]
I20260925 07:25:06.111207 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c1d0297fe024d788c848986bc3d0471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c1d0297fe024d788c848986bc3d0471" member_type: VOTER last_known_addr { host: "127.4.47.62" port: 37599 } } peers { permanent_uuid: "e6d3298e6f504d13a5c73434436f6ef3" member_type: VOTER last_known_addr { host: "127.4.47.61" port: 36245 } } peers { permanent_uuid: "f73e555988644e3ca77773c962e15c2e" member_type: VOTER last_known_addr { host: "127.4.47.60" port: 44923 } } }
I20260925 07:25:06.111341 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471 [sys.catalog]: This master's current role is: LEADER
I20260925 07:25:06.114089 27132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c1d0297fe024d788c848986bc3d0471: Generated new TSK 0
I20260925 07:25:06.114382 27132 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:25:06.143121 26943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.141467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.4.47.62:37599)
0925 07:25:06.141513 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.141516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.142401 (+ 885us) server_negotiation.cc:407] Connection header received
0925 07:25:06.142527 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.142528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.142541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.142572 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.143027 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60074: BlockingRecv error: recv got EOF from 127.0.0.1:60074 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:25:06.143961 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.141842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.4.47.61:36245)
0925 07:25:06.141880 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.141882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.143342 (+ 1460us) server_negotiation.cc:407] Connection header received
0925 07:25:06.143540 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.143541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.143553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.143584 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.143899 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36884: BlockingRecv error: recv got EOF from 127.0.0.1:36884 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 07:25:06.144753 27094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.142105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.4.47.60:44923)
0925 07:25:06.142144 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.142146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.144156 (+ 2010us) server_negotiation.cc:407] Connection header received
0925 07:25:06.144308 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.144309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.144319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.144346 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.144695 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33264: BlockingRecv error: recv got EOF from 127.0.0.1:33264 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:25:06.152333 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.151439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.4.47.61:36245)
0925 07:25:06.151497 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.151499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.151648 (+ 149us) server_negotiation.cc:407] Connection header received
0925 07:25:06.151780 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.151781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.151803 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.151838 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.152272 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36892: BlockingRecv error: recv got EOF from 127.0.0.1:36892 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 07:25:06.156324 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.155443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.4.47.61:36245)
0925 07:25:06.155483 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.155485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.155764 (+ 279us) server_negotiation.cc:407] Connection header received
0925 07:25:06.155863 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.155865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.155876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.155905 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.156256 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36894: BlockingRecv error: recv got EOF from 127.0.0.1:36894 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 07:25:06.160497 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.159618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.4.47.61:36245)
0925 07:25:06.159661 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.159663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.159924 (+ 261us) server_negotiation.cc:407] Connection header received
0925 07:25:06.160027 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.160028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.160039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.160070 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.160440 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36908: BlockingRecv error: recv got EOF from 127.0.0.1:36908 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:25:06.166646 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.165707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.4.47.61:36245)
0925 07:25:06.165751 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.165753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.166029 (+ 276us) server_negotiation.cc:407] Connection header received
0925 07:25:06.166151 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.166152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.166165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.166197 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.166573 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36918: BlockingRecv error: recv got EOF from 127.0.0.1:36918 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:25:06.177124 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.176077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.4.47.61:36245)
0925 07:25:06.176159 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.176162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.176464 (+ 302us) server_negotiation.cc:407] Connection header received
0925 07:25:06.176574 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.176575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.176586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.176617 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.177038 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36934: BlockingRecv error: recv got EOF from 127.0.0.1:36934 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 07:25:06.195503 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.194487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.4.47.61:36245)
0925 07:25:06.194534 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.194536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.194835 (+ 299us) server_negotiation.cc:407] Connection header received
0925 07:25:06.194944 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.194945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.194958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.194990 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.195441 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36938: BlockingRecv error: recv got EOF from 127.0.0.1:36938 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 07:25:06.229730 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.228911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.4.47.61:36245)
0925 07:25:06.228966 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.228969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.229136 (+ 167us) server_negotiation.cc:407] Connection header received
0925 07:25:06.229219 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.229220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.229235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.229304 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.229653 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36954: BlockingRecv error: recv got EOF from 127.0.0.1:36954 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:25:06.295743 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.295015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.4.47.61:36245)
0925 07:25:06.295148 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.295150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.295162 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:25:06.295222 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.295223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.295233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.295289 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.295676 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36964: BlockingRecv error: recv got EOF from 127.0.0.1:36964 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 07:25:06.425932 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.425026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.4.47.61:36245)
0925 07:25:06.425134 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.425136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.425147 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:25:06.425219 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.425219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.425230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.425288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.425834 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36966: BlockingRecv error: recv got EOF from 127.0.0.1:36966 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 07:25:06.684079 27019 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:25:06.683206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.4.47.61:36245)
0925 07:25:06.683297 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:06.683299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:06.683390 (+ 91us) server_negotiation.cc:407] Connection header received
0925 07:25:06.683546 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:06.683547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:06.683561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:06.683641 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:25:06.684011 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36980: BlockingRecv error: recv got EOF from 127.0.0.1:36980 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 07:25:07.041045 27122 catalog_manager.cc:1259] Loaded cluster ID: 15b2c17f56e8494981cccaae4f9aade2
I20260925 07:25:07.041170 27122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: loading cluster ID for follower catalog manager: success
I20260925 07:25:07.041985 27122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: acquiring CA information for follower catalog manager: success
I20260925 07:25:07.042330 27122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f73e555988644e3ca77773c962e15c2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:25:07.100296 27152 catalog_manager.cc:1259] Loaded cluster ID: 15b2c17f56e8494981cccaae4f9aade2
I20260925 07:25:07.100387 27152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: loading cluster ID for follower catalog manager: success
I20260925 07:25:07.101295 27152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: acquiring CA information for follower catalog manager: success
I20260925 07:25:07.101688 27152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6d3298e6f504d13a5c73434436f6ef3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:25:07.200212 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:07.197382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.4.47.61:36245)
0925 07:25:07.197512 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:07.197515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:07.197525 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:25:07.197578 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:07.197579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:07.197659 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:07.197745 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:07.197925 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.198439 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.199338 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.199557 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.199634 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:07.199824 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:07.199940 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:07.200034 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:07.200060 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 07:25:07.200110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260925 07:25:07.204676 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:07.202103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.4.47.60:44923)
0925 07:25:07.202242 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:07.202244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:07.202253 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:25:07.202320 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:07.202320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:07.202358 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:07.202426 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:07.202511 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.202973 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.203784 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.203975 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.204031 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:07.204276 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:07.204421 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:07.204500 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:07.204544 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 07:25:07.204563 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":48,"threads_started":1}
I20260925 07:25:07.209506 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:07.206510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.4.47.62:37599)
0925 07:25:07.206631 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:07.206633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:07.206867 (+ 234us) server_negotiation.cc:407] Connection header received
0925 07:25:07.206960 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:07.206961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:07.207018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:07.207156 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:07.207320 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.207810 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.208638 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.208851 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.208944 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:07.209154 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:07.209244 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:07.209313 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:07.209377 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 07:25:07.209400 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260925 07:25:07.209801 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:07.206720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.4.47.60:44923)
0925 07:25:07.206767 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:07.206769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:07.207012 (+ 243us) server_negotiation.cc:407] Connection header received
0925 07:25:07.207093 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:07.207094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:07.207104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:07.207154 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:07.207289 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.207967 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.208959 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.209249 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.209289 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:07.209495 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:07.209525 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:07.209555 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:07.209684 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 07:25:07.209712 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:25:07.209895 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:25:07.206404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.4.47.61:36245)
0925 07:25:07.206538 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 07:25:07.206540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:25:07.206690 (+ 150us) server_negotiation.cc:407] Connection header received
0925 07:25:07.206805 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:25:07.206805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:25:07.206816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:25:07.206892 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:25:07.206987 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.207798 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.208945 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:25:07.209250 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:25:07.209289 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:25:07.209494 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:25:07.209525 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:25:07.209555 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 07:25:07.209789 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 07:25:07.209813 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 07:25:07.211417 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26872
I20260925 07:25:07.216869 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 26947
I20260925 07:25:07.222105 4284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSRTLta/build/release/bin/kudu with pid 27022
2026-09-25T07:25:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2842 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5643 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270086 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST